Grain of Salt

I want to bring a post by Sean Trende at RealClearPolitics on why we should take the polling results that are being published with a grain of salt to your attention:

On a hunch, I went back and looked at the poll errors for 2013-15, and it became apparent that the errors for 2016 followed much the same pattern: They were concentrated in areas with large numbers of whites without college degrees. Indeed, the size of the poll error correlated heavily with whites-without-college-degree share (p<.001); you could explain about one-third of the difference in the size of poll miss just from knowing the share of the electorate that was whites without a college degree. We all know what happened next. Trump surprised observers by winning states that Republican presidential candidates hadn’t carried since Debbie Gibson and Tiffany fought it out for top placement in the Top 40 charts. The misses were particularly pronounced in the Midwest.

Have pollsters corrected the mistakes they made in 2016?

So, I went back and looked at the Democratic bias in the polls for swing states in 2014, 2016, and 2018. I could not use North Carolina, since there was no statewide race there in 2018. One problem I encountered is that in 2018 many states were under-polled, so RCP didn’t create an average. I’ve gone back and averaged the October polls for those states, if available (note that we don’t have three polls in October for Minnesota in 2016, hence the asterisk there). As a check on this approach, I’ve also included the error from the 538 “polls-only” model for 2018.

The results are something of a mixed bag, but overall it isn’t clear that the pollsters have really fixed the problem at all. While the bias toward Democrats was smaller in 2018 than in 2016, the bias overall was similar to what we saw in 2014, especially in the Midwest. If people remember, the polls in 2018 suggested that we should today have Democratic governors in Ohio, Iowa and Florida, and new Democratic senators in Indiana, Missouri and Florida. Obviously this did not come to pass.

Moreover, almost all of the errors pointed the same way: Republicans overperformed the polls in every Midwestern state except for Minnesota Senate/governor and Wisconsin Senate (none of which were particularly competitive).

The point here is not that the polls are intentionally biased or that we shouldn’t trust them or that we should trust them. It’s that we shouldn’t bet the farm on polls. We don’t know whether they’ll be off this year, by how much, or in which direction.

5 comments… add one
  • Drew Link

    “We don’t know whether they’ll be off this year, by how much, or in which direction.”

    No, we don’t. The diversity in results, alone, indicates the unintentional, or intentional, error. In my opinion there is also a variable not mentioned which I believe is real: the “shy respondent.”

    With all the goons out there, I wouldn’t tell a pollster anything useful.

  • Not to mention the “doxing”.

  • Drew Link


  • TarsTarkas Link

    ‘Not to mention the “doxing”.’

    These days if you’re a shy Tory in a bluish neighborhood, or even worse a shy Tory in a bluish field of employment, you’re a fool to inform ANYONE what you really think.

    I suspect we’ll see one of the biggest ‘shy Tory’ voting patterns ever this election. Which will be the justification for full-scale riots because the polls could not possibly be that wrong, the argument being that OMB and Putin must have stolen the election AGAIN.

  • Grey Shambler Link

    Got a text today from a woman who asked if I want a yard sign supporting a Republican candidate for state unicameral. My first thought was that I’ve never heard of this candidate. My second thought, as I’ve expressed before on this blog, is that I can’t afford to have my property vandalized.
    At least in this town, it’s left leaning violence I fear.

Leave a Comment