The problems with polling
Errors popped up in 2016. It happened this year too. Why?
Asking for a polling postmortem at this stage is a little bit like asking a coroner for the cause of death while the body is still at the crime scene. You’re going to have to wait to conduct a full autopsy.
But make no mistake: It’s not too early to say that the polls’ systematic understatement of President Donald Trump’s support was similar to the polling misfire of four years ago, and might have exceeded it.
After 2016, pollsters arrived at plausible explanations for why surveys had systematically underestimated Trump in battleground states. One was that state polls didn’t properly weight respondents without a college degree. Another was that there were factors beyond the scope of polling, like the large number of undecided voters who appeared to break sharply to Trump in the final stretch.
This year, there seemed to be less cause for concern: In 2020, most state polls weighted by education, and there were far fewer undecided voters.
But in the end, the polling error in states was virtually identical to the miss from 2016.
The national polls were even worse than they were four years ago, when the industry’s most highly respected and rigorous survey houses generally found Hillary Clinton leading by 4 points or less — close to her 2.1-point popular-vote victory. This year, Joe Biden is on track to win the national vote by around 5 percentage points; no major national live-interview telephone survey showed him leading by less than 8 percentage points over the final month of the race.
The New York Times/Siena College polls were also less
accurate than they were in 2018 or four years ago. In 2016, the last two Times/Siena polls were among a small group of polls to show Trump tied or ahead in Florida and North Carolina. This time, nearly all of the Times/ Siena surveys overestimated Biden to about the same extent as other surveys.
In the broadest sense, there are two ways to interpret the repeat of 2016’s polling error. One is that pollsters were wrong about what happened in 2016. As a result, the steps they took to address it left them no better off. Another is that survey research has gotten even more challenging since 2016, and whatever steps pollsters took to improve after 2016 were canceled out by a new set of problems.
Of these two, the latter interpretation may make the most sense.
“I think our polls would have been even worse this year had we employed a pre-2016 methodology,” said Nick Gourevitch of Global Strategy Group, a Democratic polling firm that took steps to better represent Trump’s sup
porters. “These things helped make our data more conservative, though clearly they were not enough on their own to solve the problem.”
The explanation for 2016’s polling error was not contrived. Many state pollsters underrepresented the number of voters without a college degree, who backed Trump in huge numbers. The pollsters went back to their data after 2016 and found that would have been much closer to the election result if they had employed the standard education adjustments that national surveys have long used.
An analysis of national surveys found that failing to weight by education cost Trump about 4 points in polling support — enough to cover much of the 2016 polling error. Other pollsters had similar findings.
But this time, education weighting didn’t seem to help.
State and national polls consistently showed Biden faring far better than Clinton did among white voters without a degree.
The results made it clear that he didn’t.
The poll results among seniors are another symptom of a deeper failure in this year’s polling. Unlike in 2016, surveys consistently showed Biden winning by comfortable margins among voters 65 and older. The final NBC/WSJ poll showed Biden up 23 points among the group; the final Times/Siena poll showed him up by 10.
So how did the polls get worse over the past four years? This is mainly speculation, but consider a few possibilities:
Trump (and the polls) hurt the polls:
There was no real indication of a “hidden Trump” vote in 2016.
But maybe there was one in 2020. For years, the president attacked the news media and polling, among other institutions. The polls themselves lost quite a bit of credibility in 2016.
It’s hard not to wonder whether the president’s supporters became less likely to respond to surveys as their skepticism of institutions mounted, leaving the polls in a worse spot than they were four years ago.
“We now have to take seriously some version of the Shy Trump hypothesis,” said Patrick Ruffini, a Republican pollster for Echelon Insights. It would be a “problem of the polls simply not reaching large elements of the Trump coalition, which is causing them to underestimate Republicans across the board when he’s on the ballot.”
(This is different from the typical Shy Trump theory that Trump supporters don’t tell pollsters the truth.)
A related possibility: During his term, Trump might have made gains among the kinds of voters who would be less likely to respond to surveys.
It’s well established that politically
Resistance hurt the polls:
engaged voters are likelier to respond to political surveys, and it’s clear that the election of Trump led to a surge of political engagement on the left. Millions attended the Women’s March or took part in Black Lives Matter protests. Progressive activists donated enormous sums and turned out in record numbers for special elections that would have never earned serious national attention in a different era.
Political pollsters have often assumed that higher turnout makes polling easier, since it means that there’s less uncertainty about the composition of the electorate. Maybe that’s not how it worked out.
Heading into the election, many surveys showed something unusual: Democrats faring better among likely voters than among registered voters.
Take Pennsylvania. The final CNN/SSRS poll of the state showed Biden up by 10 points among likely voters but by just 5 among registered voters. Monmouth showed Biden up by 7 among likely voters in a “highturnout” scenario (which it ended up being) but by 5 points among registered voters. Marist? It had a lead of 6 points among likely voters and 5 points among registered voters. The ABC/ Washington Post showed a 7point lead for Biden among likely voters and a 4-point lead among registered voters.
Turnout hurt the polls:
Remember those Times/Siena polls from October 2019 that showed Biden narrowly leading Trump? They turned out to be very close to the actual result, at least outside Florida. They were certainly closer than the Times/ Siena polls conducted since.
It wasn’t just the Times/Siena
Pandemic hurt the polls:
polls that were closer to the mark further ahead of the election. Results from pollsters in February and March look just about dead-on in retrospect, with Biden leading by about 6 points among registered voters nationwide, with a very narrow lead in the “blue wall” states, including a tied race in Wisconsin.
One possibility is that the polls were just as poor in October 2019 as in October 2020. If so, Trump actually held a clear lead during the winter. Maybe. Another possibility is that the polls got worse over the last year. And something really big did happen in American life over that time: the coronavirus pandemic.
Don’t forget the Latino vote:
There’s one state in particular where the polls were muchworse in 2020 than in 2016: Florida, where Trump made huge gains among Hispanic voters.
What happened in MiamiDade County was stunning. Biden won by just 7 points in a county where Clinton won by 29 points. No pollster saw the extent of it coming, not even those conducting polls of Miami-Dade County or its competitive congressional districts.
Most polls probably weren’t even in the ballpark. The final Times/Siena poll of Florida showed Biden with a 55% to 33% lead among Hispanic voters. In the final account, Biden may barely win the Hispanic vote in the state.
What happened in MiamiDade was not just about CubanAmericans. Although Democrats flipped a Senate seat and are leading the presidential race in Arizona, Trump made huge gains in many Hispanic communities, from the agricultural Imperial Valley and the border towns along the Rio Grande to more urban Houston or Philadelphia.