Why So Much of Polling is B.S. — F**k You, Frank!

“The numbers don’t lie” goes an oft-quoted saying… and that’s true, for the most part.  Numbers don’t lie, but what does lie is the people who are reporting the numbers.  The recent brouhaha over health care reform in the United States has brought this fact about numbers & statistics into stark relief of late.  Most people, when they read a poll, don’t really think about the numbers all that much, or they are too innumerate to really understand what they’re reading – which is how so many are easily manipulated.  And oftentimes the polls are self-contradictory.

For example, look at this recent article – which is, refreshingly, a good example of critical thinking in the modern media – concerning the question of polling public opinion on health care reform…

Health care polls leave pols dizzy

Legislators hoping to learn what their constituents think about the issue — and how to vote to keep them happy — face a dizzying deluge of hard-to-reconcile data, some of which suggests that voters are more than a little confused, as well.

What to make of it, for example, when one poll finds that 63 percent think “death panels” are a “distortion” or “scare tactic,” and only 30 percent think the issue is “legitimate,” while another finds that 41 percent believe that people would die because “government panels” would prevent them from getting the treatment they needed?

Or when one survey finds that 55 percent of Americans support the public option, while another says 79 percent favor one — but also notes that only 37 percent people surveyed actually knew what “public option” meant?

And because there is such ambiguity in these polls, those with an agenda can usually cherry-pick whatever data they want to make a case for their particular argument.  Even changing the wording of a particular question just slightly can have a huge impact…

The surveys are seemingly so sensitive that sometimes one word can spark charges of bias.

House Speaker Nancy Pelosi’s office recently griped about an NBC/Wall Street Journal poll that asked whether Americans favor a public option that would compete with private insurance companies, rather than asking how important they felt it was to have the “choice” between a public option and private insurance, as they had before.

The wording tweak left the impression that support for the public option had dropped from 76 percent to 43 percent since June, critics argued.

Others have complained about a New York Times/CBS News poll that used a word with positive associations — “Medicare” — to describe the public option.

And an ABC News summary of the results of eight polls from late July through mid-August on “the public option” found that support for a public option ranged from 43 percent to 66 percent.

So those on one side of the issue will scream about how support for a public option in health care reform is plummeting, while those on the opposite side will holler that support for a public option is stronger than ever.  And with such manipulation, who the hell knows what to think?

Perhaps the best thing to do is understand what it is that you believe, and then to advocate for your beliefs.  And when you are confronted by polling data, of any kind, you should ask yourselves the following questions:

1. Why is the poll being conducted?

2. Who is conducting the poll?  Do they have an agenda?

3. Who is being polled? Is it a representative, randomize sample, or is it biased in some way?

4. How is the poll conducted?  Is the wording of the questions meant to bias the results?

5. Is the result of the poll consistent with other reliable polls?

Let me take this opportunity to suggest that you read a wonderful book on this topic: Damned Lies & Statistics by Joel Best. It is an amazing book which very clearly outlines how various groups – politicians, media outlets, interest groups, corporations, etc – use polling & statistics in an attempt to manipulate public opinion and further their own agendas.

Perhaps the best expression of skepticism about polls was summed up by magician skeptics Penn & Teller on their show Bullshit…

In closing, let me just say… F**k You, Frank!  🙂