Problems with race and gender identification in some systems

The battle over the technology intensified last year after two researchers published a study showing bias in some of the most popular facial surveillance systems. Called Gender Shades, the study reported that systems from IBM and Microsoft were much better at identifying the gender of white men’s faces than they were at identifying the gender of darker-skinned or female faces.

Another study this year reported similar problems with Amazon’s technology, called Rekognition. Microsoft and IBM have

since said they improved their systems, while Amazon has said it updated its system since the researchers tested it and had found no differences in accuracy.

Warning that African Americans, women and others could easily be incorrectly identified as suspects and wrongly arrested, the American Civil Liberties Union and other nonprofit groups last year called on Amazon to stop selling its technology to law enforcement.

But even with improvements in accuracy, civil rights advocates and researchers warn that, in the absence of government

oversight, the technology could easily be misused to surveil immigrants or unfairly target African Americans or low-income neighborhoods. In a recent essay, Luke Stark, a postdoctoral researcher at Microsoft Research Montreal, described facial surveillance as“the plutonium of artificial intelligence,” arguing that it should be “recognized as anathema to the health of human society, and heavily restricted as a result.”

Alvaro Bedoya, who directs Georgetown University’s Center on Privacy and Technology, said more than 30 states

allow local or state authorities, or the FBI, to search their driver’s license photos.

Bedoya said these images are tantamount to being in a perpetual police lineup, as law enforcement agencies use them to check against the faces of suspected criminals.

He said the difference is that an algorithm, not a human, is pointing to the suspect. He also said comprehensive regulation of the technology is sorely lacking. “This is the most pervasive and risky technology of the 21st century,” he said.