The dangers of relying on data

Friends of mine who work in the arts and humanities have started doing something unusual, at least for them: Poring over data. This is due to the pandemic, of course. Every day, they check coronavirus case numbers, how slowly or quickly the R number is declining, and how many people in our area got vaccinated the day before. Meanwhile, social media sites are full of claims and counterclaims about all manner of other data. Is global poverty declining or increasing? What is the real level of US unemployment? The scrutiny, sometimes leading to tetchy arguments, results from people’s desire to cite — or challenge — the authority of data to support their position or worldview.

But, in other areas where data is used, there is remarkably little focus on its reliability or interpretation. One striking example concerns the “CAPTCHA” tests designed to protect websites against bots, which ask you to prove your humanity by identifying images containing common features such as boats, bicycles or traffic lights. If your choice — even if correct — differs from that of the machine system using your selection to train an image-recognition algorithm, you will be deemed inhuman.

In this example, the machine’s error is obvious. But, in other cases, it may not be possible to identify what conclusions either machine-learning systems or human analysts are drawing when they put more weight on data than the data can bear.

Many policymakers think artificial intelligence offers scope for greater cost-effectiveness and better policy outcomes. But before we entrust more decisions to data-based systems, we must be clear about the limitations of the data.

The data we use shapes our view of a complex, changing world.

But data represents reality from a particular perspective. Data of the kind deployed in policy debates is rarely completely unanchored from the world it describes, but the lens it provides can be sharp or blurry — and there is no escaping the perspective it offers.

The current hunger for databased certainty is becoming dangerous as we increasingly rely on technocratic decision procedures — including machinelearning systems — for policymaking in areas such as criminal justice, policing and welfare. Democracies often rely on constructive ambiguity to reconcile conflicting interests, such as those regarding the distribution of returns to an asset or to address the question of whether law-enforcement authorities should err on the side of imprisoning innocent people or letting criminals walk free. Claims to data-based authority minimize or eliminate the scope of ambiguity, with potentially significant consequences.

I am all in favor of more and better data, which has been essential to governments’ efforts to manage the pandemic. But the more we use data to make decisions, the more sensitive we must be to the fact that data paints an expert’s or machine’s-eye view, based on categories devised by someone who is themselves a player in society’s status game. Otherwise, we will end up with decision processes just like those rogue CAPTCHA tests — insisting that a boat is a bicycle and leaving other people with no choice but to agree.

Diane Coyle, Professor of Public Policy at the University of Cambridge, is the author, most recently, of “Markets, State, and People: Economics for Public Policy.”

©Project Syndicate