Over at CatholicCulture, Uncle Di has an interesting post about conflicting poll numbers in Massachusetts concerning which candidate is likely to win Ted Kennedy's former seat.
Last week, a Boston Globe poll of likely voters show the Democratic candidate, Martha Coakley, leading the Republican, Scott Brown, by a comfortable margin: 50- 35%.
A Public Policy Polling survey of likely voters, released the same day, showed Brown ahead, 48- 47%.
The Globe poll claimed a margin of error of +/- 4.2%; the PPP poll said its margin of error was 3.6%. Go ahead: try the numbers. They don't work.
Wait; there's a possible explanation. The Globe poll was taken January 2- 6; the PPP poll was January 7-9. So you might say that as a Little Christmas gift, Scott Brown got 13% of the likely voters.
Alternatively, you might say that there's a margin of error to the pollsters' margin of error.
Di is exactly right that there is a "margin of error" to the pollsters' margin of error–a margin that pollsters very seldom talk about.
What pollsters mean when they say that a poll has a margin of error of 4 percentage points (or whatever number) it does not mean that the true figure is really within 4 percentage points of the figure they name. They have no independent way of knowing what the true figure is. All they can do is estimate what the true figure is based on the sample of data they got.
But sometimes you get unrepresentative data, which is what the margin for error is supposed to allow for. It's a fudge factor that means, in essence, if we ran the same poll a bunch of times, the result would vary but would tend to remain within the stated margin of error.
Yet sometimes you get really unrepresentative data, and this is what pollsters don't generally point out.
In standard polling, the margin of error is based on what the result would be approximately 95% of the time, or nineteen out of twenty times. ( . . . keeping this simple so we don't have to get into standard deviations and normal distribution and confidence intervals and other technical minutiae).
So, for example, if pollsters went out and polled the right number of people to give them a 4 percent margin of error, and 48 percent of the people said that they'd vote for Candidate X then what this means is that if you re-ran the poll that nineteen out of twenty times the result you would get back should be between 44 and 52 percent, all else being equal.
But one time out of twenty the result you would get back would be wildly off, either below 44 or above 52.
So . . . bear that in mind when looking at poll numbers.
Even when the poll is properly done, one in twenty polls produces a reading so anomalous that it falls outside what the margin of error would be if you ran it another nineteen times.
On average.
We think.