Poll precision is hard to pull off

When it comes to predicting election results, polls are not that much better than the uncles holding court at mamak shops.

GE14 IS still being talked about as the result that nobody expected. The consensus the evening before the May 9 polling day was a slim win by the incumbent government, as relayed to me by various kopitiam shop-talk friends who have the inside track on all things political. This impression was further backed by the results of a poll conducted by the Merdeka Centre and released the day before elections predicting that “Barisan Nasional will prevail in the election”.

But there was one naysayer in this otherwise balmy sea of consensus. Think-tank Invoke Malaysia stepped forward two days before the election to say that Pakatan Harapan (PH) would win almost all the seats that it needed to form a government in Peninsular Malaysia, only requiring one more seat from Sabah, Sarawak or Labuan to seal the deal.

What was their basis for this confidence? Polling. They randomly called up Malaysians, asked who they would vote for, and used that as a basis for what the whole country was thinking.

Lo and behold, when election day arrived, Invoke Malaysia’s stand was vindicated. Rafizi Ramli, Invoke Malaysia’s founder (and also PKR vice-president), wrote an editorial about the victory, crediting the think tank’s successful prediction to the “empirical evidence produced by way of big data analytics and the support model that we have developed”.

Which is great. Except that Invoke didn’t get it very correct.

They predicted PH would win 111 seats in Peninsular Malaysia, when in fact the coalition only won 98 (this includes an independent candidate who later declared for PKR). These 98 seats would not have been enough if they had only won nine seats in Sabah and Sarawak, as they did in 2013.

However, this year, PKR and DAP won 16 in those states, and add to that Parti Warisan’s eight parliamentary seats in Sabah, and it all added up to a comfortable majority.

Nevertheless, the improvement in these two states did not factor in Invoke’s report predicting the PH win, and even general news coverage that those states were crucial in the final tally has been overshadowed in the heady days after the election.

How did the Merdeka Centre poll do in comparison? Basically, they under-predicted. They forecasted PH would win at least 76 seats in Peninsular Malaysia, plus a share in another 25 seats that they deemed “too close” to call (those with margins under 3% of the expected popular vote). This prediction did cover the eventual final result, but the problem is that 25 seats is a very large “error margin” to assign.

So both polls failed spectacularly, and in completely opposite directions! How could this happen? How hard is it to ask a bunch of people who they want to vote for and then calculate percentages? Well, very hard it seems.

What isn’t hard is getting enough people to poll. I find that people think that to represent a populations of millions, you need to poll lots and lots of people. Most would be surprised that polls usually make do with about 1,000.

Merdeka Centre had a sample size of 1,579, which, if it’s meant to be representative of all 15 million voters in Malaysia, would give a confidence level of about +/- 2.5%. Invoke used a sample size between 500 and 1,500 per state. For example, Selangor has the most registered voters at 2.4 million, and Invoke’s sample size of 1,112 here gives a confidence of +/- 2.9%. Both centres were at 95% confidence level.

These confidence intervals are consistent with polls done all over the world (usually +/- 3%), so sample size isn’t an issue.

I think the much bigger problem is in trying to make sure the sample is a proper representation of the larger population of likely voters. For example, if you find your poll sampled many 10-year-olds, then the poll doesn’t work because none of them will vote! This is fairly obvious, but what demographics to consider is not easy to determine. Usually people focus on age, income, education levels, race and gender.

It’s actually even more complicated than that. For example, if you know half of 20-25-year-olds don’t vote, then you might have to factor that in when calculating. Or, if you realise that half the women you ask don’t respond to your poll, then they are under-represented in your results.

How you cater for this is hard. There is something called a weighted mean where you increase a segment of the results by a multiplier to make sure everything is correctly represented. In our case, it would be like doubling our responses from women to make up for all those who didn’t answer.

As you can imagine, this is not an exact science. Neither Invoke nor Merdeka Centre give a lot of detail about how they decided the weights for their weighted means. But my belief is that there is too much subjectivity, resulting in a risk of confirmation bias (bias where you favour one piece of data or method over another because it gives the result you expect or hope for).

Another issue is the response rate, which is something that concerns American pollsters. It used to be that a third of those contacted would answer, but now that number has dropped to a tenth. With so few responding, there is a risk that the polls miss out entire segments of the population. Unfortunately, it is unclear what the response rates for the two local polls were, but a low number would be a warning flag.

It’s not to say that we shouldn’t take polls. But we need to do better in educating the public about what a poll is and what information it conveys. Right now, because Invoke correctly guessed a PH win, they are being lauded, while the Merdeka Centre is called a former government sycophant.

But I hope people will recognise that neither did very well at predicting the elections, and certainly not that much better than the uncles holding court at my local mamak shop.

Logic is the antithesis of emotion but mathematician-turned-scriptwriter Dzof Azmi’s theory is that people need both to make sense of life’s vagaries and contradictions. Write to Dzof at star2@thestar.com.my.

We need to do better in educating the public about what a poll is and what information it conveys.