Polling has evolved, and so has the Star’s approach

Long after her failed bid to be U.S. vice-president, Sarah Palin once famously — and crassly — expressed her cynical view of public opinion surveys, saying: “Polls? Nah, they’re for strippers and cross-country skiers.” Polls are certainly divisive. They’re unreliable, say critics, some of whom point to infamous flubs by polling outfits, such as when most pollsters incorrectly projected a U.S. presidential election win for Hillary Clinton in 2016.

But polls can be a useful tool when it comes to gauging public support — or lack thereof — for an elected official or candidate running for office. Public opinion surveys also provide valuable, though imperfect, snapshots and insights into how the public feels at a given point in time about a government policy proposal on the economy, on housing, food prices, transportation or the latest tax rollout.

Polling was the subject of a recent complaint against the Star that a reader brought to the National NewsMedia Council (NNC), a voluntary, self-regulatory ethics body for the news media industry in Canada.

The council deals with matters pertaining to the fairness, relevance, balance and accuracy of news and opinion reports and hears and rules on complaints — typically from the public — against members of the media.

Before I get further into the essence of the Star reader’s complaint to the council, a bit of background.

Last year, the Toronto Star entered into an informal partnership with Canadian pollster Abacus Data, a market research and strategy firm. (The Star doesn’t pay for polls.)

As part of its methodology, Abacus uses online, or so-called “opt-in” polls — different from the randomized telephone surveys commonly used in the past, in the Star and elsewhere.

According to the Star’s journalistic standards guide, when we report on poll results we must name the polling agency and include details such as the population from which the sample was drawn, the sample size, margin of error and the type of interview — telephone, mail, online, etc.

Abacus uses a nationally representative sample of Canadian adults, but unlike telephone surveys, online polls cannot be assigned a margin of error because they aren’t considered truly random samples.

(People who participate in online polls self-select or “opt-in” — so there’s a risk these polls won’t resemble the population at large, critics charge).

Participants in Abacus polls are recruited from multiple panels to reduce the bias any single panel might have. Abacus’s poll results are then “weighted” against census data to ensure their findings effectively represent the larger population.

Online polling has become a popular method for several reasons, including the fact that phone call response rates have drastically declined with consumer use of caller display.

Abacus polls are regarded as an accepted, industry-standard method of polling. For instance, polling website 338 Canada has ranked several polling operations that use online panels, including Leger, Ipsos and Abacus, identifying them among the top six most accurate in projecting election results.

338 Canada, which uses statistical modelling to assess the accuracy of polls, reached this conclusion based on examining polling projections from 18 Canadian elections for the past decade (two federal elections and provincial elections from all 10 provinces) and compared them to actual election results.

There are studies that have compared opt-ins to telephone surveys. For example, one study from BMC, an organization with a portfolio of 300 peerreviewed journals that share discoveries from research communities in the fields of science, technology, engineering and medicine, compared the findings from surveys of 1,003 people questioned by telephone in Quebec to 956 queried online on the topic of Lyme disease.

“Findings showed that neither the telephone nor the web panel modes of data collection can be considered more representative of the target population” was their conclusion.

But other studies strongly suggest online polling can be less personalized and lead to less detailed responses. Some studies of opt-in polls have found bogus respondents — bots — or instances where respondents just answer “yes” regardless of the questions.

Abacus has tools to help prevent bots from infiltrating its surveys. The company also seeks to ensure respondents are thoughtfully answering by including several questions to gauge whether participants are actually engaged in the research.

Over time, the Star has taken the view that random telephone polls are not necessarily more reliable. So, we don’t consider using Abacus a degradation of our standards, but rather a realization that the world of polling has changed.

The perfect poll — randomized or opt-in — is not available anywhere.

“Technology has fundamentally changed how polling works and our journalism must adapt to this new reality,” says the Star’s editor-in-chief, Anne Marie Owens. “This also means that it’s more important than ever that we be transparent about pollsters’ methodologies and their limits and that we take care never to overstate the significance of any single poll.”

The Star reader who complained to the news media council took issue with us publishing a Canadian Press article that used an opt-in poll by Leger. The CP article included this explanation: A total of 1,632 Canadian respondents participated in the web survey, along with 1,002 Americans. It cannot be assigned a margin of error because online polls are not considered truly random samples. This is akin to the disclaimer wording the Star uses when it publishes opt-ins.

Part of the reader complaint that randomized telephone polls should be used by media instead of opt-ins, is the latter aren’t assigned a degree of accuracy — a margin of error — and are therefore “subject to manipulation by special interest groups who may be promoting the polls.”

In dismissing the reader complaint, the council said in part: “the NNC finds no grounds to support the view that reporting on a poll without a margin of error is in violation of the (Star’s) guidelines, so long as the relevant context is provided to readers.”

At the Star we’ll be updating our guidelines to account for the use of online polls.