An article published in the New York Times last week, titled “What’s the Matter with Polling?” cited recent high profile election polls that turned out to be wrong, and identified a myriad of reasons why the polling business is in trouble. Although the article focused specifically on election polling, the fundamental problems the piece cited are applicable to virtually all types of surveys, including market research.
The difficulties are not actually all that new. It’s been at least 10, and probably more like 15 years where survey organizations had to deal with huge social and technological changes such as the growth of cell phones; declining response rates due to people’s busy lives and increased privacy concerns; non-scientific Web based surveys (you can read a spirited debate on the credibility of social surveys in the July/August issue of The Information Advisor's Guide to Internet Research, the journal that I co-edit), and the challenge of capturing valid data when measuring anything related to the fast moving digital world. But the article does speak to an acceleration of these trends.
But what worries me though is something else, something even bigger.
It seems that we are simply not trusting polls anymore. More people dismiss them out of hand as biased, meaningless, or gamed to achieve a pre-determined outcome. (You can get a sense of some of these attitudes by reading the comments section of the Times' article.)
Some of this cynicism is justified and healthy. But my concern is the knee jerk rejection of even well conducted surveys as if these were simply another “opinion” or just BS. This attitude can dovetail with the other distressing phenomenon whereby people dismiss science and data when the results do not agree with what they believe, expect, or want them to be.
I think this attitude may be particularly true for young people. The GenX and Millennial generations lived during a time when so many traditional institutions have shown themselves to be less trustworthy and thereby are now seen as less credible. Furthermore, younger people have grown up in a media environment when fact, opinion and increasingly, advertising, are mixed together and conflated.
But just as some reject evolution, human induced climate change, or the science showing no link between vaccinations and autism, and as others refuse to answer questions from the Bureau of the Census as a form of governmental ‘intrusion” of one’s privacy, the same cynical or uninformed attitudes appear to be seeping into a general rejection of all surveys and polls as well.
That is too bad of course, as well conducted surveys provide valuable snapshots on what we all or a designated sub-population believe, think, and act and can serve as one important input to guide decisions ranging from determining public policy to introducing a new product.
What to do?
I think it’s going to be up to the survey industry’s trade associations and specific polling and market research organizations to tackle these challenges head on. This means much more transparency and education in how surveys and polls are conducted and more focus on enhancing information literacy for the public. People need to understand matters like sample size, representativeness, and get a few basic primers on probability too—for example, helping the public understand what that "plus/minus confidence level" really means—and that forecasts do not mean that there is a 100% probability that the forecast event will occur.
From a business researcher’s—or any researcher’s perspective—this is one more meta problem going on with “what’s the matter” with polling. For more information on how to best understand surveys and polls, I recommend the site: "Polling Fundamentals and Concepts: An overview for Journalists" created by the Journalists Resource site at Harvard University's Shorenstein Center on Media, Politics and Public Policy, and available here:
The difficulties are not actually all that new. It’s been at least 10, and probably more like 15 years where survey organizations had to deal with huge social and technological changes such as the growth of cell phones; declining response rates due to people’s busy lives and increased privacy concerns; non-scientific Web based surveys (you can read a spirited debate on the credibility of social surveys in the July/August issue of The Information Advisor's Guide to Internet Research, the journal that I co-edit), and the challenge of capturing valid data when measuring anything related to the fast moving digital world. But the article does speak to an acceleration of these trends.
But what worries me though is something else, something even bigger.
It seems that we are simply not trusting polls anymore. More people dismiss them out of hand as biased, meaningless, or gamed to achieve a pre-determined outcome. (You can get a sense of some of these attitudes by reading the comments section of the Times' article.)
Some of this cynicism is justified and healthy. But my concern is the knee jerk rejection of even well conducted surveys as if these were simply another “opinion” or just BS. This attitude can dovetail with the other distressing phenomenon whereby people dismiss science and data when the results do not agree with what they believe, expect, or want them to be.
I think this attitude may be particularly true for young people. The GenX and Millennial generations lived during a time when so many traditional institutions have shown themselves to be less trustworthy and thereby are now seen as less credible. Furthermore, younger people have grown up in a media environment when fact, opinion and increasingly, advertising, are mixed together and conflated.
But just as some reject evolution, human induced climate change, or the science showing no link between vaccinations and autism, and as others refuse to answer questions from the Bureau of the Census as a form of governmental ‘intrusion” of one’s privacy, the same cynical or uninformed attitudes appear to be seeping into a general rejection of all surveys and polls as well.
That is too bad of course, as well conducted surveys provide valuable snapshots on what we all or a designated sub-population believe, think, and act and can serve as one important input to guide decisions ranging from determining public policy to introducing a new product.
What to do?
I think it’s going to be up to the survey industry’s trade associations and specific polling and market research organizations to tackle these challenges head on. This means much more transparency and education in how surveys and polls are conducted and more focus on enhancing information literacy for the public. People need to understand matters like sample size, representativeness, and get a few basic primers on probability too—for example, helping the public understand what that "plus/minus confidence level" really means—and that forecasts do not mean that there is a 100% probability that the forecast event will occur.
From a business researcher’s—or any researcher’s perspective—this is one more meta problem going on with “what’s the matter” with polling. For more information on how to best understand surveys and polls, I recommend the site: "Polling Fundamentals and Concepts: An overview for Journalists" created by the Journalists Resource site at Harvard University's Shorenstein Center on Media, Politics and Public Policy, and available here:
RSS Feed
