The racism of technology - and why driverless cars could be the most dangerous example yet
There is a rule for dealing with computers: garbage in, garbage out. Put the wrong number of zeroes in your Excel spreadsheet and it will unthinkingly pay your staff pennies on the pound; train a self-driving car to recognise human figures by showing it millions of pictures of white people, and it might struggle to identify pedestrians of other races.
That was the finding of researchers from Georgia Tech, who analysed how effective various “machine vision” systems were at recognising pedestrians with different skin tones. The results were alarming: AI systems were consistently better at identifying pedestrians with lighter skin tones than darker. And not by a little bit: one headline comparison suggests that a white person was 10% more likely to be correctly identified as a pedestrian than a black person.
Self-driving cars are by no means the first technology to fail when confronted by other ethnicities: Google’s image-recognition system notoriously failed to discern black people from gorillas. Almost every product design has failed to grapple with the reality of humanity, from Kodak colour film that reduced dark skin to a pitch-black smudge; to motion-activated taps and driers that refuse to acknowledge the presence of a brown hand but will trigger for a white one.
A 10-tonne driverless truck poses a higher penalty for error, however. The good news is that most actually existing self-driving cars use more than one type of sensor, including several that do not rely on visible light at all: Tesla cars, for instance, have a radar built in to the front of the vehicle, while Google’s Waymo uses a bulky, but extraordinarily accurate Lidar system instead; think radar but with lasers.
The bad news is that there is strong market pressure to move towards camera-only systems because of the huge cost savings. Such systems would only hit the streets in large numbers if they proved significantly safer than human drivers, but even that raises the important question: safer for whom?