In an op-ed at the Washington Post pollster David Hill explains why telephone opinion polling has reached the end of its usefulness:
When I first undertook telephone polling in the early 1980s, I could start with a cluster of five demographically similar voters — say, Republican moms in their 40s in a Midwestern suburb — and expect to complete at least one interview from that group of five. I’d build a sample of 500 different clusters of five voters per cluster, or 2,500 voters total. From that number, I could be reasonably assured that 500 people would talk to us. The 500 clusters were designed to represent a diverse cross-section of the electorate.
As the years drifted by, it took more and more voters per cluster for us to get a single voter to agree to an interview. Between 1984 and 1989, when caller ID was rolled out, more voters began to ignore our calls. The advent of answering machines and then voicemail further reduced responses. Voters screen their calls more aggressively, so cooperation with pollsters has steadily declined year-by-year. Whereas once I could extract one complete interview from five voters, it can now take calls to as many as 100 voters to complete a single interview, even more in some segments of the electorate.
And here’s the killer detail: That single cooperative soul who speaks with an interviewer cannot possibly hold the same opinions as the 99 other voters who refused.
In short, we no longer have truly random samples that support claims that poll results accurately represent opinions of the electorate.
Instead, we have samples of “the willing,” what researchers call a “convenience sample” of those consenting to give us their time and opinions. Despite knowledge of this, pollsters (including myself) have glossed over this reality by dressing up our results with claims of polls having a “margin of error” of three or four percentage points when we knew, or should have known, that the error factor is incalculable given the non-random sample. Most pollsters turned to weighting results to “fix” variations in cooperation, but this can inadvertently amplify sampling errors due to noncooperation.
That strikes me as about right. Combine answering machines, caller ID, cellphones, and just plain distrust of political polling and telephone polling is pretty much a thing of the past as much as rotary phones are.
That doesn’t mean that people will stop trying to figure out how people are planning to vote or that data analysis is going to go away. Far from it. I think that the incentives are sufficiently high that both will continue. I just think that it will take different people with different skills and the identification of better proxy measures to do the job.
But if telephone polling isn’t dead, it’s certainly pining for the fjords.