Surveys are not entertainment

This paper recently asked a question about a survey it conducted on its web site. They got more responses than usual, and while they didn’t quite come out and say so, I’m guessing the result of that one was surprising. So, what do these survey results mean?

As someone who conducts surveys as part of my job, I can tell you. The results mean absolutely nothing. Perhaps I shouldn’t mince words: the survey results from a newspaper web site are not worth the pixels they’re not printed on.

Here’s the thing: there is a science to survey work. Last fall, after the unexpected election result, I wrote a column defending the polling industry. People think they got it all wrong when in fact, the vast majority of the polls were within the margin of error. The thing is, in a close election, it doesn’t take much error at all for the polls to be seen as “wrong.” The problem last November wasn’t so much the polls themselves as the horrible interpretation of them with many pundits brazenly stating that now-president Trump had no chance. The data did not justify that even when the polls did seem to slightly favor Clinton. In fact, that interpretation could easily have cost Clinton the election if it caused just a few thousand of her supporters to stay home. Survey work is not easy. There are a huge number of details to consider, from question wording to even the order of the questions. One that is important though is the issue of sampling. If one takes a random sample of several hundred people representing the population whose opinion you want to measure, the results you get should be within a few percentage points of the opinion of the population being sampled.

The point is, even with declining survey response rates in recent decades, there is a science to the practice and surveys are usually pretty accurate when done right.

There are many ways to bias a sample. It can happen accidentally. I did it myself once several years ago. We wanted a random sample of the college’s students, but not to duplicate many classes being surveyed. Someone suggested sampling the English writing classes. It seemed reasonable at the time.

The problem was, English classes alone are not a random sample of college students. The bias we introduced was that we got more remedial students than a random sample would have. The next time we conducted the same survey, the results weren’t truly comparable.

A survey on a web page violates almost every precept of sampling. It’s not close to random.

First of all, to get to the survey, one must visit the web page in question. In the case of a city like Porterville, that means having internet access, which will mean getting wealthier Portervillians than a random sample would.

People accessing the web site may also be those who have a subscription to the paper or who at least are interested in viewing something on it that day. There are many ways that would be non-random. Newspapers are read by older than average people for example. On a particular day, those interested might be those who visit for a particular reason, such as a popular or trending article. Then, they have to care enough about the issue being surveyed, to cast their ‘vote.’

Then there are intentional biases, such as attempts to influence the results. It is common for newspaper surveys to go viral online because they influence (rather than measure) public opinion. People will ask their friends and other like-minded people to vote. In one survey a couple of years ago, conducted by the Orlando Business Journal about the controversial film Blackfish which was critical of Seaworld’s animal care practices, more than half the responses came from one IP address, which turned out to be owned by Seaworld.

Then, there is the opportunity to vote twice. Even if a newspaper limits the number of votes, a user who wants to manipulate the results can do so, simply by switching browsers, using a private browsing session, or deleting her or his browser cookies.

Media sources explain away biased survey results by noting on an online survey that it is “not a scientific poll,” terminology that much of the public does not understand. Or worse yet, they state that the survey is “for entertainment purposes only.”

What they’re really telling you is what I just have: any results you see are absolutely meaningless. They bear little resemblance to the reality of public opinion.

I’m probably picky as a research professional, but I don’t find misleading people entertaining at all. I find it irresponsible. Ignore these every time.