What questions should you ask when you hear a claim based on data?
With cases, deaths, reproduction numbers, opinion polls and more, we get bombarded with statistics every day. But how can you spot a naughty number, a shabby statistic or dubious data? Lists of questions have been given by Tim Harford, Dave and Tom Chivers, and in The Art of Statistics (which David wrote), with considerable overlap as they grapple with the same essentials. Here is the short list that we use ourselves.
The first question: how trustworthy is the source of the story? Are they honest, competent and reliable or are
they trying to manipulate my emo
tions, say by provoking anxiety or reassurance? Are they giving the whole picture or selecting bits to suit themselves? Like the review that found insects had declined, but it turned out had only searched for studies that showed a decline.
Second: how reliable is the statistic? A count may not be counting what we think it is: Covid cases only represent those who have symptoms and decide to have a PCR test and are becoming increasingly distinct from infections. Survey estimates seek to generalise from a sample to a target population, but have both margins of error and possible systematic biases, while model outputs reflect assumptions and selected scenarios. The uncertainty and the quality of the underlying evidence should be communicated with due humility.
Third: how trustworthy is any claim being made on the basis of the data? Context is all; a “100% increased risk” may make the headlines, but twice not very much is still not very much. Historical statistics can provide a guide to whether something really is a big number. Differences may be attributed to cause rather than just correlation, since observational analyses can have systematic biases or lingering confounders. And sometimes there is basic misunderstanding of good data; ONS found that, among people testing positive, vaccination increased the chance
someone would be Omicron rather than Delta, but this led to the claim that vaccination increased the risk of testing positive.
Statistics enlighten our uncertain world, helping people make decisions.
We need thoughtful consideration, not reflexive cynicism, when we see claims derived from statistical evidence.
• David Spiegelhalter is chair of the Winton Centre for Risk and Evidence Communication at Cambridge. Anthony Masters is statistical ambassador for the Royal Statistical Society