Sunday, June 25, 2006

That question sucks!

Andrew Gelman commented yesterday on a recent CBS News Poll which asked the following question:
"Should U.S. troops stay in Iraq as long as it takes to make sure Iraq has a stable democracy, even if it takes a long time, or leave as soon as possible, even if Iraq is not completely stable?"
Gelman points out that it's a "double-barrelled" question with
"... the assumption that U.S. troops will 'make sure Iraq has a stable democracy,' along with the question of how long the troops should stay".
He also notes that the New York Times piece on the poll included "a yucky graph (as Tufte would put it, 'chartjunk')", which I have shown here.

It really doesn't get much better than this: an exceedingly slanted question and an exceedingly silly graph! If I may, I'd like to name it the bow tie graph ... but is that taken? On this, I defer to Kaiser over at Junk Charts. Dressing up simple percentages with multicoloured variable-sized triangle regions is ingenious, but misguided. It's a shame that so much creative effort is misspent. There are many situations where new ideas for displaying data are needed, but instead we get a never-ending stream of bizarre ways to display percentages.

The question itself stands as striking evidence of media bias. Respondents should have been given a third option: "That question sucks!" (I'm reminded of the segment on This Hour Has 22 Minutes called "That Show Sucked!") As it is, it appears that about 7% of Republicans, 5% of Democrats, and 8% of Independents didn't answer the question. Some of those who didn't answer may have felt the question was too stupid to dignify a response. But it's quite interesting to speculate on what the responses might have been like if the following (perhaps more dignified) choice had been included:
I can't answer this question because I believe it has built-in assumptions that I disagree with.
I often find that this is my reaction to opinion poll questions. To be fair, even with an honest effort to understand people's views (which seems utterly implausible in the case above), it's not easy to ask good questions, or to provide a good set of response choices. This seems like a strong argument in favour of qualitative research methods, which avoid imposing a predefined (and possibly ideologically loaded) structure on responses. I'm very much in the quantitative camp, but for complex matters like political opinions, I can see that there may be some value, particularly in the early stages of research, in taking a qualitative approach.

When it comes to exposing ideological bias, no one's better than Tom Tomorrow. Check out this archive of cartoons from his brilliant series, This Modern World.
Bookmark and Share


Blogger Dave said...

The July edition of Macleans has their "Canada Survey" - which I found to be quite flawed, especially their questions around the "character" of Quebec as compared to the rest of the country. Made me think about your "built in assumptions" choice...

12:06 PM, June 29, 2006  

Post a Comment

<< Home