Is Joe Biden behind? The problem with polls
Without polls, you can’t understand politics, but with them, you can misunderstand a lot. With both major party presidential nominations sewn up, we’re deep into the season in which fretting over polls can become an obsession.
That’s especially true this year, as former President Donald Trump holds a small but persistent edge over President Joe Biden in most national and swing-state surveys. That’s led many Democrats to search deep into the innards of polls in an often self-deluding search for error.
The fact is, polls continue to get election results right the vast majority of the time. At the same time, errors do exist, often involving either problems collecting data or troubles interpreting it.
Let’s examine a non-political example.
A Holocaust myth?
In December, the Economist published a startling poll finding: “One in five young Americans thinks the Holocaust is a myth,” the headline said. Fortunately for the country, although perhaps not for the publication, it’s the poll finding that may have been mythical.
In January, the nonpartisan Pew Research Center set out to see if it could replicate the finding. They couldn’t. Pew asked the same question the Economist poll asked and found that the share of Americans ages 18 to 29 who said the Holocaust was a myth was not 20%, but 3%.
The problem isn’t a bad pollster: YouGov, which does the surveys for the Economist, is among the country’s most highly regarded polling organizations. But the methodology YouGov uses, known in the polling world as opt-in panels, can be victimized by bogus respondents. That may have been the case here.
Panel surveys are a way to solve a big problem pollsters face: Very few people these days will answer phone calls from unknown numbers, making traditional phone-based surveys extremely hard to carry out and very expensive.
The problem with panels
Rather than randomly call phone numbers, polling organizations can solicit thousands of people who will agree to take surveys, usually in return for a small payment. For each survey, the pollsters select people from the panel to make up a sample that’s representative of the overall population.
Some people join simply for the money, however, then may speed through, answering questions more or less at random. Previous research by Pew has found that such bogus respondents most often claim to belong to groups that are hard to recruit, including young people and Latino voters.
Pollsters have found evidence of organized efforts to infiltrate panels, sometimes involving “multiple registrations from people who are outside the U.S.,” Douglas Rivers, the chief scientist at YouGov and a political science professor at Stanford, wrote in an email. Those could be efforts to bolster particular causes or candidates or, more often, schemes to make money by collecting small sums over and over again.
“We have a whole host of procedures to screen out these panelists,” Rivers wrote, adding that the firm was continuing to analyze what happened with the Holocaust question.
Be skeptical
There’s a takeaway in all this for people interested in politics, especially in a hotly contested election year: Don’t over-focus on any individual poll, especially if it has a startling finding that hasn’t cropped up anywhere else.
Be skeptical about sweeping conclusions about events that are still unfolding. And even, or maybe especially, when a poll shows your favored candidate trailing, take it for what it is — neither an oracle, nor a nefarious plot, but a snapshot in time.