You wouldn’t consult an e-cigarette manufacturer’s website if you were doing research on the hazards of electronic cigarettes, right? Or rely on data on the growth of solar energy from an organization whose sole mission is to advance its cause. Your natural, logical—and largely sound—instinct would be to avoid using information provided by entities who have a vested interest in achieving a particular outcome.
Kind of obvious, right?
But I recently did choose to use data and analysis from an advocacy group—one that I strongly disagreed with no less.
My project was to research and find out how many Americans were eschewing driving cars and choosing public transportation instead. I came across a comprehensive survey created by the American Public Transit Association (APTA) that found that Americans took a record 10.8 billion trips in 2013, "the highest annual ridership in 58 years” the subtitle of the report read. According to its President and CEO the survey illustrated “a fundamental shift going on in the way we move about our communities…”
Then I found some additional commentary, on a blog that challenged the APTA’s notion that transit use was truly increasing. I quickly scanned down the page to find the source and discovered that it was The Heartland Institute—an organization with a “libertarian and conservative” agenda to “promote public policy based on individual liberty, limited government, and free markets.” I was, in fact, already familiar with and quite unsympathetic towards this group. For years it has actively been trying to convince the public to discount human created climate change, as well as pursue other politically “on the right” points of view that I personally do not ascribe to.
So my immediate reaction was to dismiss this blog piece as just more propaganda. But I decided to suspend judgment a bit, and see what else my research turned up and revisit the site further down the road. So later when it was time to pull together my research, I did take the time back to read the full blog piece. And what I found was in fact a convincing argument.
The reasons that I found the piece convincing were:
Heartland’s argument itself was actually quite straightforward: while it was true that transit ridership was at an all-time high as the APTA data showed, that organization failed to put its data into two very important pieces of context: changes in population and mass transit use by one particular city. On the first point, the blog provided reputable data that between 2008-2013 the total US population rose from 304 million to 316 million. Once that data is taken into account, annual public transportation use, on a per person basis, actually falls from about 35 trips to 34. Furthermore, if one takes out New York City’s contribution—which represents 40% of all transit riders—American public transit use on a per capita basis actually falls for those years.
But although this new information was extremely valuable in putting the APTA data into the missing and necessary context, it still is just one more piece of data and still does not tell the “whole story.” Further research uncovered larger, macro trends supporting likely ongoing growth of public transportation in the U.S. due to forces like aging boomers looking to move to urban centers with nearby businesses; millennials lack of passion and interest in driving; and a renewed increase among certain cities towards building new light rail and public transport options, among other relevant trends. The President and CEO’s claim that Americans were in fact fundamentally shifting their way of travel looked to be correct, but not because of the total rise of transit use found in its survey.
Kind of obvious, right?
But I recently did choose to use data and analysis from an advocacy group—one that I strongly disagreed with no less.
My project was to research and find out how many Americans were eschewing driving cars and choosing public transportation instead. I came across a comprehensive survey created by the American Public Transit Association (APTA) that found that Americans took a record 10.8 billion trips in 2013, "the highest annual ridership in 58 years” the subtitle of the report read. According to its President and CEO the survey illustrated “a fundamental shift going on in the way we move about our communities…”
Then I found some additional commentary, on a blog that challenged the APTA’s notion that transit use was truly increasing. I quickly scanned down the page to find the source and discovered that it was The Heartland Institute—an organization with a “libertarian and conservative” agenda to “promote public policy based on individual liberty, limited government, and free markets.” I was, in fact, already familiar with and quite unsympathetic towards this group. For years it has actively been trying to convince the public to discount human created climate change, as well as pursue other politically “on the right” points of view that I personally do not ascribe to.
So my immediate reaction was to dismiss this blog piece as just more propaganda. But I decided to suspend judgment a bit, and see what else my research turned up and revisit the site further down the road. So later when it was time to pull together my research, I did take the time back to read the full blog piece. And what I found was in fact a convincing argument.
The reasons that I found the piece convincing were:
- The data it used to back up its points were from neutral and respected sources, including the Bureau of the Census, and the article linked back to those original sources.
- The argument that disputed the conclusions of the APTA study was logical, clear, and easy to understand.
- The tone of the piece itself was informative, and did not call on, refer to, or mix up its larger value related viewpoints with the data and argument.
Heartland’s argument itself was actually quite straightforward: while it was true that transit ridership was at an all-time high as the APTA data showed, that organization failed to put its data into two very important pieces of context: changes in population and mass transit use by one particular city. On the first point, the blog provided reputable data that between 2008-2013 the total US population rose from 304 million to 316 million. Once that data is taken into account, annual public transportation use, on a per person basis, actually falls from about 35 trips to 34. Furthermore, if one takes out New York City’s contribution—which represents 40% of all transit riders—American public transit use on a per capita basis actually falls for those years.
But although this new information was extremely valuable in putting the APTA data into the missing and necessary context, it still is just one more piece of data and still does not tell the “whole story.” Further research uncovered larger, macro trends supporting likely ongoing growth of public transportation in the U.S. due to forces like aging boomers looking to move to urban centers with nearby businesses; millennials lack of passion and interest in driving; and a renewed increase among certain cities towards building new light rail and public transport options, among other relevant trends. The President and CEO’s claim that Americans were in fact fundamentally shifting their way of travel looked to be correct, but not because of the total rise of transit use found in its survey.
So for me, the lessons were, yes, to remain skeptical, but not closed to data presented by an advocacy group (even those I strongly dislike!) and that while a piece of data may be sound, the interpretation may not be. It’s also a reminder that there are many ways to look at a particular phenomenon, and each organization will slice and present it in the way that is going to serve their interests the best.
That’s why any good researcher will find and evaluate multiple sources of information—including data from advocacy groups—and ultimately make his or her own conclusions.
That’s why any good researcher will find and evaluate multiple sources of information—including data from advocacy groups—and ultimately make his or her own conclusions.
RSS Feed
