Accuracy of Political Polls
In this election season, we are constantly bombarded by the latest polling data. Often, polls taken by different pollsters on the same day yield wildly different results. Some of this is to be expected, but it is pretty easy to sort through the accuracy of all the pollsters and determine which ones are the most accurate. I have undertaken this task and learned some pretty interesting stuff.
First, my methodology… I analyzed poll data from the 2004 Presidential election, 2004 Senate elections, and 2006 Gubernatorial elections in as many states as I could find. This included well over 1000 different individual polls. I included only the final polls from each pollster and only those taken in the last 30 days prior to the election in question. I then compared each of these to the result to determine if the poll within the margin of error and how far off was the poll from the actual result.
Among the national pollsters, there are four which are clearly the most accurate based on how often they were within the margin of error.
- Quinnipiac 84%
- Mason-Dixon 80%
- Rasmussen 72%
- Research 2000 68%
Note that the best pollster is only within the margin of error 84% of the time.
Some national pollsters are just down right awful.
- Survey USA 59%
- CNN 54%
- Zogby 50%
That’s right, with these you have a coin flip (or slightly better) chance of being within the margin of error, which means these polls, on the whole, are basically useless.
Perhaps not surprisingly, the national pollsters significantly out-perform regional and local pollsters, like local newspapers, state agencies, etc. Regional polls have been within the margin of error only 50% of the time (again, a coin flip, and pretty much useless) while national polls, as a whole, have been within the margin of error 67% of the time.
How far off are most polls? Well, to be honest, pretty far off. If we just consider the national polls, the average poll is off by 3.48 points one way or the other, just within the margin of error of most polls. Taking just the top four pollsters listed above drops that to 2.78 points.
Also, there are several websites that do statistical analysis on polls and compute their own “poll.” These can be analyzed, too. The only one for which I could find old data is Real Clear Politics. Performing the same analysis on their numbers, you see that they are within the margin of error 75% of the time with an average error of 2.95 points, about the same as the best polls.
So, what can we learn from this? Well, we can all but ignore regional and local polls and certain national polls and focus more closely on just a select few polls or just the “combination polls” like RCP’s. In this election, it appears to confirm that this thing is a dead heat.
There’s more to come, including analysis of whether polls are biased towards one party, whether certain states are easier to poll than others, and projections of the upcoming presidential race based on my analysis.
Click here to bookmark this page.
HUH……I read your last sentence of this article and then went to look at he polls you say are best. Obama is WAY ahead in the polls you say are best. Is this a really old article or are you trying the Limbaugh/Hannity spin? I realize this is a conservative sight, but at least state the facts.
Thanks for the comment!
Part of the answer is because the article was posted almost 2 weeks ago, and the polls have moved in Obama’s favor during that period of time. Part of the answer is that these polls have a margin of error of +/-4%, meaning that a 4% lead by either candidate is, statistically, a tie; enough of these polls are within that margin to call this a dead heat. Again, over the two weeks this article was written, things have moved to where Obama is, statistically speaking, slightly ahead.
NR
[...] data in several key battleground states. Let’s take a look at the most accurate polls (Quinnipiac, Rasmussen, and Mason-Dixon by my analysis) in some of the key [...]