The numbers don’t lie? Oh yes, they do

As politicians try to ‘follow the science’, Simon Ings reveals the ugly truth about bad data and systemic rot

LCALLING BULLS--by Jevin D West and Carl T Bergstrom 336pp, Allen Lane, £20, ebook £9.99 SCIENCE FICTIONS by Stuart Ritchie 368pp, Bodley Head, £18.99, ebook £9.99

ast week I received a press release headlined “One in four Brits say ‘No’ to Covid vaccine”. This was such staggeringly bad news, I decided it couldn’t possibly be true. And sure enough, it wasn’t.

Armed with the techniques taught to me by biologist Carl Bergstrom and data scientist

Jevin West, I “called bulls---” on this unwelcome news, which, after all, bore all the hallmarks of clickbait. For a start, the question on which the poll was based was badly phrased. On closer reading, it turns out that 25 per cent would decline if the Government “made a Covid-19 vaccine available tomorrow”. Frankly, if it were offered tomorrow, I’d be a refusenik myself. I prefer my medicines tested first.

But what of the real meat of the claim – that daunting figure of 25 per cent? It turns out that a sample of 2,000 was selected from a sample of 17,000 drawn from the self-selecting community of subscribers to a lottery website. But hush my cynicism: I am assured that the sample of 2,000 was “within +/-2 per cent of ONS quotas for age, gender, region, socio-economic grade, and 2019 vote, using machine learning”.

In other words, some effort has been made to make the sample of 2,000 representative of the UK population (but only on five criteria, which is not very impressive. And that whole “+/-2 per cent” business means that up to 40 of the sample weren’t representative of anything). No amount of “machine learning” can massage away the fact that the sample was too thin to serve the sweeping conclusions drawn from it (“Only one in five Conservative voters – 19.77 per cent – would say No” – it says, to two decimal places) and is anyway drawn from a non-random population. Mention of the miracle that is artificial intelligence is almost always a bit of prestidigitation to veil the poor quality of the original data.

Exhausted yet? Then you may well find Calling Bulls--- essential reading. Even if you feel you can trudge through verbal bulls--easily enough, this book will give you the tools to swim through numerical snake-oil. And this is important, because numbers easily slip past the defences we put up against mere words. Bergstrom and West teach a course at the University of Washington from which this book is largely drawn, and hammer this point home in their first lecture: “Words are human constructs,” they say; “Numbers seem to come directly from nature.” Shake off your naive belief in the truth or naturalness of the numbers quoted in news stories and advertisements, and you’re halfway towards knowing when you’re being played.

Say you diligently applied the lessons in Calling Bulls---, and really came to grips with percentages, causality, selection bias and all the rest. You may well discover that you’re now ignoring everything – every bit of health advice, every overwrought Nasa announcement about life on Mars, every economic forecast, every exit poll. Internet pioneer Jaron Lanier reached this point last year when he came up with Ten Arguments for Deleting Your Social Media

Accounts Right Now. More recently, the bestselling Swiss pundit Rolf Dobelli has ordered us to Stop Reading the News.

Both deplore our current economy of attention, which values online engagement over the provision of actual information (as when, for instance, a review like this one is headlined “These two books about bad data will break your heart”; instead of being told what the piece is about, you’re being sold the promise of an emotional experience).

Bergstrom and West believe that public education can save us from this torrent of micro-manipulative blither. Their book is a handsome contribution to that effort. We’ve lost Lanier and Dobelli, but maybe the leak can be stopped up. This, essentially, is what the authors are about; they’re shoring up the Enlightenment ideal of a civic society governed by reason.

Underpinning this ideal is science, and the conviction that the world is assembled on a bedrock of truth, fundamentally unassailable truth. Philosophical nitpicking apart, science undeniably works. But in Science Fictions, Stuart Ritchie, a psychologist at King’s College, shows just how shoddy the whole business can get. He has come to praise science, not to bury it; nevertheless, his analyses of science’s current ethical ills – fraud, hype, negligence and so on – are devastating.

The sheer number of problems besetting the scientific endeavour becomes somewhat more manageable once we work out which ills are institutional, which have to do with how scientists communicate, and which are existential problems that are never going away whatever we do.

Our evolved need to express meaning through stories is an existential problem. Without stories, we can do no thinking worth the name, and this means that we are always going to prioritise positive findings over negative ones, and find novelties more charming than rehearsed truths. Such quirks of the human intellect can be and have been corrected by healthy institutions at least some of the time over the past 400-odd years. But our unruly mental habits run wildly out of control once they are harnessed to a media machine driven by attention. And the blame for this is not always easily apportioned: “The scenario where an innocent researcher is minding their own business when the media suddenly seizes on one of their findings and blows it out of proportion is not at all the norm,” writes Ritchie.

It’s easy enough to mount a defence of science against the tin-foil-hat brigade, but Ritchie is attempting something much more discomforting: he’s defending science against scientists. Fraudulent and negligent individuals fall under the spotlight occasionally, but institutional flaws are Ritchie’s chief target.

This is a devastating analysis of science’s ethical ills – fraud, hype, negligence

Reading Science Fictions, we see field after field fail to replicate results, correct mistakes, identify the best lines of research, or even begin to recognise talent. In Ritchie’s proffered bag of solutions are desperately needed reforms to the way scientific work is published and cited, and some more controversial ideas about how international megacollaborations may enable science to catch up on itself and check its own findings effectively (or indeed at all, in the dismal case of economic science).

At best, these books together offer a path back to a civic life based on truth and reason. At worst, they point towards one that’s at least a little bit defended against its own bulls---. Time will tell whether such efforts can genuinely turn the ship around, or if they are simply here to entertain us with a spot of deckchair juggling. But there’s honest toil here, and a lot of smart thinking with it. Reading both, I was given a fleeting, dizzying reminder of what it once felt like to be a free agent in a factual world.

Seemingly ‘natural’, numbers slip past the defences we put up against mere words