Lack of city rules for AI risks ‘biased outcomes’

Several city government agencies risk “biased outcomes” due to a lack of guidelines for their use of artificial intelligence, according to a new state comptroller’s report.

The audit, issued Thursday morning by state Comptroller Tom DiNapoli, focused primarily on four city agencies: the NYPD, the Administration for Children’s Services and the Education and Buildings departments.

It concluded that the city isn’t assessing the effectiveness of its AI tools or setting discernable accuracy standards, “leaving it vulnerable to misguided, inaccurate or biased outcomes in several programs that can directly impact New Yorkers’ lives,” the comptroller’s report said.

“Government’s use of artificial intelligence to improve public services is not new,” DiNapoli said. “But there needs to be formal guidelines governing its use, a clear inventory of what’s being used and why, and accuracy standards, so that the algorithms that can help train educators, identify potential criminal suspects, prioritize child abuse cases and inspect buildings don’t do more harm than good.”

The most recently available city data show that, as of 2021, city agencies reported using 21 “algorithmic tools,” which include those employing artificial intelligence or machine learning techniques. Examples of such tools include the NYPD’s use of facial recognition technology, which has come under fire for flaws when it comes to race and ethnicity, and ACS’s use of technology aimed at predicting which open child-abuse cases are most likely to lead to future and severe harm.

According to DiNapoli’s report, the city doesn’t have an “effective AI governance framework,” and that while agencies are required to report their use of artificial intelligence, there’s little in the way of guardrails around how to actually use the technology.