I wish I knew what Nate Silver’s opinion on Patrick Ruffini’s recent findings on opinion polls was. The short version is that opinion polls overestimated Democratic electoral performance: over the last four cycles, he says 73% of the polls he examined overstated the Democrat, with the effect particularly pronounced in summer polling. Nate has apparently expressed an opinion but, unfortunately, it’s hidden behind a paywall.
Based on my admittedly anecdotal experience canvassing Chicago wards, I think there’s a relatively benign possible explanation at least compared with some of the explanations being offered. I’ve gone door to door in multiple Chicago precincts accompanied by the Republican ward committeeman and my experience has been that Republican households are less likely than Democratic households to answer the door and engage with a canvasser. If a similar difference exists in willingness to respond to pollsters, that alone would introduce a partisan response bias.
The recent Median Strategies debacle raises a related but rather different question: are polls as reliable as they used to be? If you’re not familiar with Median Strategies the organization admittedly published false opinion polls as a “social experiment”. How one might distinguish between a social experiment and political shenanigans is unclear to me. I only pay attention to polls from a small number of outlets: my old client NORC, Gallup, Pew Research, a few others. There’s an enormous amount of disinformation these days.
Be very careful out there.
I am not sure how much is deliberate misinformation. Certainly I think it occurs with people being paid by the political parties. That said, I think that polling in general has become more difficult and less reliable. People get hassled about participating in polls frequently and many/most people now try to avoid them. But because there is demand for poll results polling companies have proliferated and since they want to make money they have resorted to a lot of internet surveys which are notoriously unreliable. There also way too many with small sample sizes. As I have taught medical students for years, if you want guarantee studies with bad results (or ones you want to find) do lots of small studies. As an aside that happened a lot during covid and it’s how people decided a lot of treatments like Ivermectin work.
So like you I stick with the large, older polling outfits who have the expertise and funding to do it right. I also largely ignore polling for volatile events that have low rates of participation like primary elections.
Steve
This observation if true undermines a fundamental assumption of polling.
The mathmatics of polling require the sample to be representative of the whole population on the question being asked. With low response rates, pollsters have to weight responses to match the population at large in demographic profile. But that weighting process requires the differential response rates to polls to be from factors independent or non-casual to the question being asked — i.e. Weighing for how many “Republicans” to have in a poll is impossible if being “Republican” causes a person to be less likely to answer a poll vs a demographic fact like whether they went to college, work in a blue collar vs white collar job, etc. is the cause.
One of the things that people routinely miss about low turnout elections (like primaries) is that with a low turnout practically anything can happen—we shouldn’t draw sweeping conclusions from the results.
Along with that I’ve been stunned that when you dig into the reports of high turnout for a primary frequently the writer often means “high for a primary”. One of those high turnouts was 25%.
Some of this appears to be that Democrats are relatively more engaged with politics earlier. There’s a little back-and-forth here bw/ Ruffini and Nate, in which emphasizes midterm polls correct at the end, while presidential polls do not:
https://x.com/PatrickRuffini/status/2090257171841708488
Not sure why that would be.
Thanks, PD. I hope the “dialogue” continues.