NYC measure aims to rein in AI hiring tools employers use

Job candidates rarely know when hidden artificial intelligence tools are rejecting their resumes or analyzing their video interviews. But New York City residents could soon get more say over the computers making behind-thescenes decisions about their careers.

A bill passed by the City Council in early November would ban employers from using automated hiring tools unless a yearly bias audit can show they won’t discriminate based on an applicant’s race or gender. It would also force makers of those AI tools to disclose more about their opaque workings and give candidates the option of choosing an alternative process — such as a human — to review their application.

Proponents liken it to another pioneering New York City rule that became a national standard-bearer earlier this century — one that required chain restaurants to slap a calorie count on their menu items.

Instead of measuring hamburger health, though, this measure aims to open a window into the complex algorithms that rank the skills and personalities of job applicants based on how they speak or what they write. More employers, from fast-food chains to Wall Street banks, are relying on such tools to speed up recruitment, hiring and workplace evaluations.

“I believe this technology is incredibly positive but it can produce a lot of harms if there isn’t more transparency,” said Frida Polli, co-founder and CEO of New York startup Pymetrics, which uses AI to assess job skills through game-like online assessments. Her company lobbied for the legislation, which favors firms like Pymetrics that already publish fairness audits.

But some AI experts and digital rights activists are concerned that it doesn’t go far enough to curb bias, and say it could set a weak standard for federal regulators and lawmakers to ponder as they examine ways to rein in harmful AI applications that exacerbate inequities in society.

“The approach of auditing for bias is a good one.

The problem is New York City took a very weak and vague standard for what that looks like,” said Alexandra Givens, president of the Center for Democracy & Technology. She said the audits could end up giving AI vendors a “fig leaf ” for building risky products with the city’s imprimatur.

Givens said it’s also a problem that the proposal only aims to protect against racial or gender bias, leaving out the trickier-to-detect bias against disabilities or age. She said the bill was recently watered down so that it effectively just asks employers to meet existing requirements under U.S. civil rights laws prohibiting hiring practices that have a disparate impact based on race, ethnicity or gender. The legislation would impose fines on employers or employment agencies of up to $1,500 per violation — though it will be left up to the vendors to conduct the audits and show employers that their tools meet the city’s requirements.

The Council voted 38-4 to pass the bill Nov. 10, giving a month for Mayor Bill De Blasio to sign or veto it or let it go into law unsigned.

MARY ALTAFFER/AP Dr. Frida Polli, co-founder and CEO of Pymetrics, demonstrates AI technology used to assess job skills. Her company lobbied for legislation passed by New York City.