Humans can’t escape accountability for AI

Algorithms are in theory useful tools, but their ‘black box’ decisions shouldn’t be accepted without question

The way organisations make decisions is changing. An explosion in the volume of data, coupled with the growing sophistication and accessibility of algorithms, means that organisations have increasing opportunities to use machine learning and artificial intelligence to support decisionmaking.

This is a good thing, in theory. Take recruitment. It has never been physically easier to apply for a job – a few clicks of a button online, and you’re in the running. As a result, there has been a surge in the number of employment applications. However, this hasn’t been great for ensuring applicants are treated consistently.

Tired eyes skim-reading hundreds of CVS was never going to be a recipe for fair decision-making: if you happen to be near the bottom of the pile, good luck. But AI is beginning to make things easier. More importantly, the technology could also make things fairer – there is enormous potential for data-driven tools to help standardise processes and address areas of discretion (or indiscretion) where human biases can creep in.

And yet we have seen many examples of algorithms amplifying historic biases, or creating them anew. In the US, for example, an algorithm used to predict the likelihood of a criminal going on to reoffend was shown to have a bias against black defendants: white defendants were more likely to be incorrectly judged as low risk, and black defendants more likely to be incorrectly judged by the algorithm as high risk.

We can, and must, do better. We have to be clearer on accountability: the buck stops at the door of leaders. Organisations and individuals – be they in the public or private sector – need to be clear that they retain accountability for decisions made by both their human teams and their algorithms. Until we have this, leaders in organisations are less likely to push for answers about whether an algorithm is working properly, and how it reached its conclusions. Simply accepting a “black box” without justification won’t cut it. This is particularly crucial in the public sector: citizens can’t opt out of local government or interactions with the police, and decisions made in these sectors have real-life, acutely felt impacts.

This requires good, anticipatory governance. Many of the high-profile cases of algorithmic bias could have been anticipated with careful evaluation and mitigation of the potential risks. Organisations, and more specifically their leaders, need to ensure that the right capabilities and structures are in place to make certain this happens both before algorithms are introduced into decision-making processes, and throughout those processes. Doing this well requires listening, to get an understanding of the expectations and concerns of the people who will be using an algorithm, and those who will be affected by it.

Leaders must strive to get this right now. There are steps for regulators, government and industry to take – but if we collectively drag our heels, public confidence will dwindle.

Part of the challenge is in skills: we need to build an ecosystem of skilled professionals to help organisations get fairness right and provide AI assurance. The Government has asked the Centre for Data Ethics and Innovation, which publishes its report today on this issue, to focus on AI assurance as a priority. We are bringing together a diverse range of organisations to begin the work needed to accelerate the development of an effective AI accountability system in the UK. Think vehicle safety, but for AI: we ensure the cars on our roads are safe to drive, and new cars are becoming safer year after year – so how will we do the same for AI? This is no small task, but it is achievable.

Enabling data and AI technologies to drive better, fairer, more trustworthy decision-making is a challenge that countries face around the world. By taking a lead in this area, and working together with our international partners, the UK can help to address bias and inequalities, not only within our own borders, but also across the globe.