The problems with polling
Errors popped up in 2016. It happened this year too. Why?
Asking for a poll ing postmortem at this stage is a little bit like asking a coroner for the cause of death while the body is still at the crime scene. You’re going to have to wait to conduct a full autopsy.
But make no mistake: It’s not too early to say that the polls’ systematic understatement of President Donald Trump’s support was similar to the polling misfire of four years ago, and might have exceeded it.
After 2016, pollsters arrived at plausible explanations for why surveys had systematically underestimated Trump in battleground states. One was that state polls didn’t properly weight respondents without a college degree. Another was that there were factors beyond the scope of polling, like the large number of undecided voters who appeared to break sharply to Trump in the final stretch.
This year, there seemed to be less cause for concern: In 2020, most state polls weighted by education, and there were far fewer undecided voters.
But in the end, the polling error in states was virtually identical to the miss from 2016.
The national polls were even worse than they were four years ago, when the industry’s most highly respected and rigorous survey houses generally found Hillary Clinton leading by 4 points or less — close to her 2.1-point popular-vote victory. This year, Joe Biden is on track to win the national vote by around 5 percentage points; no major national live-interview telephone survey showed him leading by less than 8 percentage points over the final month of the race.
The New York Times/ Siena College polls were also less accurate than they were in 2018 or four years ago. In 2016, the last two Times/ Siena polls were among a small group of polls to show Trump tied or ahead in Florida and North Carolina. This time, nearly all of the Times/Siena surveys overestimated Biden to about the same extent as other surveys.
In the broadest sense, there are two ways to interpret the repeat of 2016’s polling error. One is that pollsters were wrong about what happened in 2016. As a result, the steps they took to address it left them no better off. Another is that survey research has gotten even more challenging since 2016, and whatever steps pollsters took to improve after 2016 were canceled out by a new set of problems.
Of these two, the latter interpretation may make the most sense.
“I think our polls would have been even worse this year had we employed a pre-2016 methodology,” said Nick Gourevitch of Global Strategy Group, a Democratic polling firm that took steps to better represent Trump’s supporters. “These things helped make our data more conservative, though clearly they were not enough on their own to solve the problem.”
The explanation for 2016’s polling error was not contrived. Many state pollsters underrepresented the number of voters without a college degree, who backed Trump in huge numbers. The pollsters went back to their data after 2016 and found that would have been much closer to the election result if they had employed the standard education adjustments that national surveys have long used.
An analysis of national surveys found that failing to weight by education cost Trump about 4 points in polling support — enough to cover much of the 2016 polling error. Other pollsters had similar findings.
But this time, education weighting didn’t seem to help.
State and national polls consistently showed Biden faring far better than Clinton did among white voters without a degree.
The results made it clear that he didn’t.
The poll results among seniors are another symptom of a deeper failure in this year’s polling. Unlike in 2016, surveys consistently showed Biden winning by comfortable margins among voters 65 and older. The final NBC/WSJ poll showed Biden up 23 points among the group; the final Times/Siena poll showed him up by 10.
In the final account, there will be no reason to believe any of it was real.
Polling clearly has some serious challenges. The industry has always relied on statistical adjustments to ensure that each group, like white voters without a degree, represents its proper share of the sample.
But this helps only if the respondents you reach are representative of those you don’t. In 2016, they seemed to be representative enough for many purposes. In 2020, they were not.
So how did the polls get worse over the past four years?
This is mainly speculation, but consider a few possibilities:
Trump (and the polls) hurt the polls:
There was no real indication of a “hidden Trump” vote in 2016.
But maybe there was one in 2020.
For years, the president attacked the news media and polling, among other institutions. The polls themselves lost quite a bit of credibility in 2016.
It’s hard not to wonder whether the president’s supporters became less likely to respond to surveys as their skepticism of institutions mounted, leaving the polls in a worse spot than they were four years ago.
“We now have to take seriously some version of the Shy Trump hypothesis,” said Patrick Ruffini, a Republican pollster for Echelon Insights. It would be a “problem of the polls simply not reaching large elements of the Trump coalition, which is causing them to underestimate Republicans across the board when he’s on the ballot.”
(This is different from the typical Shy Trump theory that Trump supporters don’t tell pollsters the truth.)
A related possibility: During his term, Trump might have made gains among the kinds of voters who would be less likely to respond to surveys, and might have lost additional ground among voters who would be more likely to respond to surveys. College education, of course, is only a proxy for the traits that predict whether someone might back Trump or respond to a poll. There are other proxies as well, like whether you trust your neighbor; volunteer your time; are politically engaged.
Another proxy is turnout: People who vote are likelier to take political surveys. The Times/Siena surveys go to great lengths to reach nonvoters, which was a major reason our surveys were more favorable for the president than others in 2016. In 2020, the nonvoters reached by The Times were gener
ally more favorable for Biden than those with a track record of turning out in recent elections.
Re s i sta n ce h u r t t h e polls:
It’s well established that politically engaged voters are likelier to respond to political surveys, and it’s clear that the election of Trump led to a surge of political engagement on the left. Millions attended the Women’s March or took part in Black Lives Matter protests. Progressive activists donated enormous sums and turned out in record numbers for special elections that would have never earned serious national attention in a different era.
This surge of political participation might have also meant that the resistance became likelier to respond to political surveys, controlling for their demographic characteristics. Are the “MSNBC moms” now excited to take a poll while they put Rachel Maddow on mute in the background? Like most of the other theories presented here, there’s no hard evidence for it — but it does fit with some well-established facts about propensity to respond to surveys.
Turnout hurt the polls:
Political pollsters have often assumed that higher turnout makes polling easier, since it means that there’s less uncertainty about the composition of the electorate. Maybe that’s not how it worked out.
Heading into the election, many surveys showed something unusual: Democrats faring better among likely voters than among registered voters.
Take Pennsylvania. The final CNN/SSRS poll of the state showed Biden up by 10 points among likely voters but by just 5 among registered voters. Monmouth showed Biden up by 7 among likely voters in a “highturnout” scenario (which it ended up being) but by 5 points among registered voters. Marist? It had a lead of 6 points among likely voters and 5 points among registered voters. The ABC/ Washington Post showed a 7-point lead for Biden among likely voters and a 4-point
lead among registered voters.
It’s still too soon to say whether Republican turnout beat Democratic turnout, but it’s possible.
In Florida, the one state where we do have hard turnout data, registered Republicans outnumbered registered Democrats by about 2 percentage points among those who actually voted, even though Democrats outnumber Republicans among registered voters by about 1.5 points in the state. Here, there is no doubt that the turnout was better for the president than the polls suggested, whether they’re private polls or the final Times/ Siena poll — which showed registered Republicans with an edge of 0.7 points.
If Trump fared better among likely voters than among registered voters in Pennsylvania, a fundamental misfire on the estimate of turnout could quickly explain some of the miss.
Unlike the other theories presented here, this one can be proved false or true. States will eventually update their voter registration files with a record of whether voters turned out in the election. We’ll be able to see the exact composition of the electorate by party registration, and we’ll also be able to see which of our respondents voted.
Pandemic hurt the polls:
Remember those Times/Siena polls from October 2019 that showed Biden narrowly leading Trump? They turned out to be very close to the actual result, at least outside Florida. They were certainly closer than the Times/Siena polls conducted since.
It wasn’t just the Times/ Siena polls that were closer to the mark further ahead of the election. Results from pollsters in February and March look just about deadon in retrospect, with Biden leading by about 6 points among registered voters nationwide, with a very narrow lead in the “blue wall” states, including a tied race in Wisconsin.
One possibility is that the polls were just as poor in October 2019 as in October
2020. If so, Trump actually held a clear lead during the winter. Maybe. Another possibility is that the polls got worse over the last year. And something really big did happen in American life over that time: the coronavirus pandemic.
“The basic story is that after lockdown, Democrats just started taking surveys, because they were locked at home and didn’t have anything else to do,” said David Shor, a Democratic pollster who worked for the Obama campaign in 2012. “Nearly all of the national polling error can be explained by the post-COVID jump in response rates among Dems.”
Don’t forget the Latino vote:
There’s one state in particular where the polls were much worse in 2020 than in 2016: Florida, where Trump made huge gains among Hispanic voters.
What happened in Miami-Dade County was stunning. Biden won by just 7 points in a county where Clinton won by 29 points. No pollster saw the extent of it coming, not even those conducting polls of MiamiDade County or its competitive congressional districts.
Most polls probably weren’t even in the ballpark. The final Times/Siena poll of Florida showed Biden with a 55% to 33% lead among Hispanic voters. In the final account, Biden may barely win the Hispanic vote in the state.
What happened in Miami-Dade was not just about Cuban-Americans. Although Democrats flipped a Senate seat and are leading the presidential race in Arizona, Trump made huge gains in many Hispanic communities, from the agricultural Imperial Valley and the border towns along the Rio Grande to more urban Houston or Philadelphia.
Many national surveys don’t release results for Hispanic voters because any given survey usually has only a small sample of the group. It will be some time until the major pollsters post their results to the Roper Center, a repository of detailed polling data.