The Skeptical Teacher

Musings of a science teacher & skeptic in an age of woo.

Why So Much of Polling is B.S. — F**k You, Frank!

Posted by mattusmaximus on October 3, 2009

“The numbers don’t lie” goes an oft-quoted saying… and that’s true, for the most part.  Numbers don’t lie, but what does lie is the people who are reporting the numbers.  The recent brouhaha over health care reform in the United States has brought this fact about numbers & statistics into stark relief of late.  Most people, when they read a poll, don’t really think about the numbers all that much, or they are too innumerate to really understand what they’re reading – which is how so many are easily manipulated.  And oftentimes the polls are self-contradictory.

For example, look at this recent article – which is, refreshingly, a good example of critical thinking in the modern media – concerning the question of polling public opinion on health care reform…

Health care polls leave pols dizzy

Legislators hoping to learn what their constituents think about the issue — and how to vote to keep them happy — face a dizzying deluge of hard-to-reconcile data, some of which suggests that voters are more than a little confused, as well.

What to make of it, for example, when one poll finds that 63 percent think “death panels” are a “distortion” or “scare tactic,” and only 30 percent think the issue is “legitimate,” while another finds that 41 percent believe that people would die because “government panels” would prevent them from getting the treatment they needed?

Or when one survey finds that 55 percent of Americans support the public option, while another says 79 percent favor one — but also notes that only 37 percent people surveyed actually knew what “public option” meant?

And because there is such ambiguity in these polls, those with an agenda can usually cherry-pick whatever data they want to make a case for their particular argument.  Even changing the wording of a particular question just slightly can have a huge impact…

The surveys are seemingly so sensitive that sometimes one word can spark charges of bias.

House Speaker Nancy Pelosi’s office recently griped about an NBC/Wall Street Journal poll that asked whether Americans favor a public option that would compete with private insurance companies, rather than asking how important they felt it was to have the “choice” between a public option and private insurance, as they had before.

The wording tweak left the impression that support for the public option had dropped from 76 percent to 43 percent since June, critics argued.

Others have complained about a New York Times/CBS News poll that used a word with positive associations — “Medicare” — to describe the public option.

And an ABC News summary of the results of eight polls from late July through mid-August on “the public option” found that support for a public option ranged from 43 percent to 66 percent.

So those on one side of the issue will scream about how support for a public option in health care reform is plummeting, while those on the opposite side will holler that support for a public option is stronger than ever.  And with such manipulation, who the hell knows what to think?

Perhaps the best thing to do is understand what it is that you believe, and then to advocate for your beliefs.  And when you are confronted by polling data, of any kind, you should ask yourselves the following questions:

1. Why is the poll being conducted?

2. Who is conducting the poll?  Do they have an agenda?

3. Who is being polled? Is it a representative, randomize sample, or is it biased in some way?

4. How is the poll conducted?  Is the wording of the questions meant to bias the results?

5. Is the result of the poll consistent with other reliable polls?

Let me take this opportunity to suggest that you read a wonderful book on this topic: Damned Lies & Statistics by Joel Best. It is an amazing book which very clearly outlines how various groups – politicians, media outlets, interest groups, corporations, etc – use polling & statistics in an attempt to manipulate public opinion and further their own agendas.

Perhaps the best expression of skepticism about polls was summed up by magician skeptics Penn & Teller on their show Bullshit

In closing, let me just say… F**k You, Frank!  🙂

5 Responses to “Why So Much of Polling is B.S. — F**k You, Frank!”

  1. badrescher said

    That is one of my favorite books (although it’s been a long time since I read it). May I also recommend a “positive statistics” book? One which shows the way in which properly conducted and interpreted statistics can be used to learn all sorts of cool things?

    Freakonomics: A Rogue Economist Explores the Hidden Side of Everything by Steven D. Levitt and Stephen J. Dubner

    Oh, yeah, and I would not trust a poll conducted by ANYONE, but political polls? Dung of the stinkiest kind.

  2. teacherninja said

    Fuck You, Frank!

    I like Nate Silver, though…

  3. Jim said

    I think a distinction can (and should) be made between issue polling and polling on a political race. In the aggregate (a la NAte Silver, as mentioned above) polling on a race is usually a fairly accurate representation of what will happen.

    One could argue that this has some self-reinforcing properties, and they’d be right, but that’s a different question altogether.

    • mattusmaximus said

      Good point. And, for the record, I agree that Nate Silver’s website is probably one of the best in dissecting & analyzing polls.

    • badrescher said

      There should also be a distinction between statistics and reporting of statistics.

      I remember once reading a book criticizing politicians and political pundits. Ironically, the author made the statement that a particular state was the 48th worst state in the nation for something. Sounds really bad. Until you realize that means they were the 3rd BEST!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: