Not all polls are created equal

Pollsters are clearly facing problems reaching a representative sample of the public

They’re great for entertainment purposes, but then again, so are fortune tellers and horoscopes.

If you’re looking at public opinion polls to predict things, well, there are wins and there are losses. And lately, more losses.

We’ve got to start looking at polling differently - because though some of the tools get better all the time, the results are not the gospel we pretend they are.

Two polls in the past week or so highlights that for me: a Toronto poll suggesting Doug Ford could be competitive in a race against Toronto Mayor John Tory, and a Corporate Research Associates poll about marijuana use in the Atlantic provinces.

For the Toronto poll, I’d question why the numbers were even used.

Certainly, the media shares some of the blame: in a world of short-staffed newsrooms, the “instant news” of free polling results is a ready-made page-filler. Heck, the pollsters often essentially write the story for you.

But putting polls up in news stories - even warning, like the CBC did with the Doug Ford poll, that the numbers might be off - can skew results. The Ford poll was wonky enough that the CBC said, “CBC Toronto is not publishing the full polling results because of a number of concerns raised by its internal research department, primarily that the poll lacks a randomized sample.”

That being said, they published it anyway.

The poll contacted people by robocall - 233,640 people. Only 15,576 Torontonians actually answered the calls and, really, you have to wonder how representative that subset of people would actually be. (I think the only empirical evidence you can take from that is people hate robocalls.)

Then, there’s CRA’s marijuana poll. Right now, marijuana use is illegal. I don’t know about anyone else, but if someone called me out of the blue said they were calling from a polling agency, and asked me about future marijuana use, I’d certainly be more than circumspect in the way I answered.

Numbers for prospective post-legalization marijuana users across the Atlantic provinces in the poll looked like this: 15 per cent in P.E.I., 19 per cent in Nova Scotia, 23 per cent in Newfoundland and Labrador, and 20 per cent in New Brunswick.

I’ll be very interested in what the numbers are actually like, post-legalization.

Pollsters are clearly facing problems reaching a representative sample of the public: regular telephone numbers are shrinking fast, and cellphones don’t have easy, geographical searchable listings. Web-based polling is often a crapshoot - like firing a shotgun into a fogbank and hoping for ducks.

The recent Calgary municipal election saw polling company Mainstreet forecast that candidate Bill Smith would win by between nine and 17 percentage points. Smith actually lost, with Naheed Nenshi taking eight per cent more votes; Mainstreet had called Smith’s election a virtual certainty.

Being off by 23 per cent is a real kick in the teeth.

It’s fine to go back after the fact and dissect what went wrong, and even apologize for it, as Mainstreet did after the Calgary election. Mainstreet also used robocalls to sample people, but found after the election that it skewed the sample away from young people, and away from voters who didn’t speak English as a first language.

Polling companies do free polls and release them to the media to promote and advertise their own business - they do political polls often for the very people who are running.

If the numbers are wrong, the damage is often already done people hear the numbers, and think it’s science. And sometimes, those polls persuade people to put their support behind a cause that’s nothing more than a sampling illusion.

I’d say simply, buyer beware. Nineteen times out of 20.

“People hear the numbers, and think it’s science.”