AI has become our judge, jury and jailer

Children marked as ‘future criminals’, a taxi driver mysteriously threatened – this exposé is terrifying

320pp, Picador, T £16.99 (0808 196 6794), RRP£20, ebook £9.99

In the summer of 2016, a woman in Amsterdam received a letter from the mayor’s office. It informed her that not only had one of her sons – who had often had trouble with the police – been placed on a list of violent criminal youths, but her other son had been put on a separate database of teenagers who might become criminals in due course.

The second list had been compiled with the help of ProKid, a machine-learning system designed by academics in conjunction with Dutch police. In a scenario reminiscent of Philip K Dick’s novella The Minority Report, ProKid used data as wide-ranging as a young person’s address, their relationships, whether they had been arrested (not charged or convicted), whether they had been a “victim or witness of a crime”, and whether they had even played truant from school, to predict whether children aged between 12 and 18 would become involved with crime.

The premise was that if the state could intervene in each child’s life now, through social services, psychologists and engagement at school, it could prevent that happening.

In reality, ProKid often had the opposite effect. Children couldn’t understand why they had been stigmatised, and often turned to crime precisely because they felt the authorities already saw them as criminals. The mother of the 16-year-old in Amsterdam calls it “a crazy system”, and tells the journalist Madhumita Murgia: “It messed his life up.”

As Murgia shows in Code Dependent, her wide-ranging and mildly terrifying book on the effects of artificial intelligence, there’s often little explanation of how an AI system makes such decisions, or how to change its rulings. A freedom-ofinformation request revealed that even Amsterdam’s city authorities “had little idea why the algorithmic system had chosen specific families over others”. These were “opaque systems whose internal workings weren’t fully explainable even by their architects”.

Code Dependent is a penetrating look at how we’re allowing artificial intelligence to infiltrate all parts of society, from policing and welfare to justice and health, to the point where whole lives are being altered – often ruined – by systems that hardly any of us understand. Most of the advancements and complexities of AI are being generated in Silicon Valley, but Murgia, who is the Financial Times’s first artificialintelligence editor, deliberately omits this area, and she skips the finer technical details, too. This book is about ordinary humans, especially poorer or more marginalised groups around the world, who are already “living in the shadow of AI”. Their stories form 10 deft chapters that show how machine-learning models are affecting everything from (in Murgia’s titles) “Your Rights” and “Your Identity” to “Your Body” and “Your Livelihood”.

Again and again, we see people confronted with the nightmare of trying to find out why an AI system has made a life-changing decision against them. In the chapter “Your Boss”, we hear from an Uber driver in London who was sent automated messages from the ride-hailing company accusing him of fraud and threatening to terminate his contract. Yet no one at Uber appears to know why its AI system has decided this, nor do they have any further help to provide. As the driver tells Murgia: “They said over and over, ‘The system can’t be wrong, what have you done?’” This, Murgia argues, is the predicament of many workers “in an algorithmic age”: you can be suspended “at a moment’s notice, at the will of an AI system”, with little redress.

AI, on the other hand, is also likely to improve our lives in countless ways that we don’t yet know. Murgia agrees, writing at the end of Code Dependent that she hopes AI will help humans “live our best and happiest lives” – though if she believes that, you might ask why she hasn’t included more evidence to support it.

Her most uplifting chapter is about doctors in a rural part of western India using an AI app called qTrack to determine quickly whether a patient has tuberculosis.

When Uber’s AI accuses a driver of fraud, no one at the company knows why

“It really helps to confirm your thoughts,” says one doctor. “It is reassuring, like a colleague.” Yet apps such as these cost money, which has to be provided by the state, else they become accessible only to the wealthy – something that, in fact, is already happening. Murgia notes that in a private hospital in Delhi, a sign by the reception desk asks visitors: “Do you want to be diagnosed by AI for tuberculosis? This way for a VIP service.”

So: what hope? Code Dependent looks at those kicking back against the power of the algorithms, including some exceptionally brave activists in China who are risking their freedom to protest against dystopian facial-recognition cameras. At the end, Murgia also offers 10 questions we should consider when we encounter an AI tool. For instance, what are the channels by which we can opt out of an AI system? In deeply consequential areas, such as criminal justice or welfare, who will be accountable for the decisions made by AI?

These are commendable guidelines; yet, even so, Murgia leaves you feeling that there’s unlikely to be an opt-out overall. As she warns us at the opening of her book, AI is “altering the very experience of being human”. There may be precious little we can do about that.