Damn Lies & Statistics: Who’s Counting?
Posted by mattusmaximus on April 18, 2009
In the United States, tax day (April 15th) has just passed, and this year we had a rather interesting series of anti-tax events organized by Fox News called “Tea Parties”. The organizers of these events were, at one point, estimating perhaps millions of people in attendance.
To me, the interesting thing about events like this is the numbers of people that are reported to have attended. In an attempt to apply a version of the argument from popularity, organizers of such events will tend to over-estimate attendance while those in opposition to such events usually try to downplay those same numbers.
Sometimes, when we see polls or statistics being reported in the media or by some interest group, the numbers are presented to us as what Dr. Joel Best (author of “Damned Lies & Statistics”) refers to as “little nuggets of truth”. However, a deeper analysis of such reports often presents a very different picture.
To use the Fox News Tea Parties as a timely example, some reporters on Fox News were stating publicly that they wouldn’t be surprised if the attendance went into the millions. Critics of Fox News, such as some commentators on the MSNBC network, report considerably lower numbers. But how can they report different numbers of the same event? It all comes down to how you do the counting.
Whenever a poll or set of statistics is presented to you, you should ask yourself three questions:
1. Who is asking the questions?
2. Who is being polled or asked questions?
3. How are those questions being asked?
Based upon the answers to these three questions, you can get a pretty decent idea on whether or not the numbers being reported are trustworthy.
Here’s a good example of what I mean – a clip from Penn & Teller’s show Bullshit! (episode: Numbers – season 4), which illustrates just how easy it is for polls & statistics to be manipulated to further an agenda…
So, back to our example of the Tea Parties, what are we to make of all these various crowd estimates? The most objective source I’ve found to date on this is Nate Silver’s FiveThirtyEight.com website. Silver states there…
Here are the new and revised listings; followed by a complete list from top to bottom. The new listings bring the cumulative estimate of attendance to 311,460 between 346 cities. The same caveats apply as before: although I’ve included any estimates I’ve found that seem even reasonably nonpartisan and credible, there were many protests in which reliable crowd estimates were not readily available or where there wasn’t even any press coverage at all. However, essentially all major cities and state capitals should now be accounted for.
In closing, I’ll reference Nate Silver’s final comments on this point. They are very objective and help to put things into a bit of perspective.
I present this data merely as a reference; it is not intended as a value judgment about the success of the protests or their lack thereof. One should bear in mind that ours is a large country, and that gatherings of this magnitude (if several hundred individual protests can be thought of as a “gathering”) are not especially uncommon. For instance, protests in favor of immigration reform drew several million participants in the spring of 2006, including several individual events of at least 300,000. Likewise, anti-war protests in 2003 involved attendance of at least 300,000 in a single American city (New York) on a single day.
There you go. I leave it to the reader to draw their own conclusions, as any good skeptic should. Just remember, the next time you see any poll or a report of some set of statistics, it isn’t as cut and dry as some would have you think. You have to ask yourself this basic question: Who’s counting?