Jacob Weisberg

The New York Review of Books - - Contents - Jacob Weisberg 1 See Ju­lia Ang­win and Terry Par­ris Jr., “Face­book Lets Ad­ver­tis­ers Ex­clude Users by Race,” ProPublica, Oc­to­ber 28, 2016; and Ju­lia Ang­win, Ari­ana Tobin, and Madeleine Varner, “Face­book (Still) Let­ting Hous­ing Ad­ver­tis­ers Ex­clude Users by

Al­go­rithms of Op­pres­sion: How Search En­gines Re­in­force Racism by Safiya Umoja Noble Au­tomat­ing Inequal­ity: How High-Tech Tools Pro­file, Po­lice, and Pu­n­ish the Poor by Vir­ginia Eubanks

Al­go­rithms of Op­pres­sion: How Search En­gines Re­in­force Racism by Safiya Umoja Noble. NYU Press, 229 pp., $28.00 (pa­per)

Au­tomat­ing Inequal­ity: How High-Tech Tools Pro­file, Po­lice, and Pu­n­ish the Poor by Vir­ginia Eubanks. St. Martin’s, 260 pp., $26.99 1.

In May 2018, a new data and pri­vacy law will take ef­fect in the Euro­pean Union. The prod­uct of many years of ne­go­ti­a­tions, the Gen­eral Data Pro­tec­tion Reg­u­la­tion is de­signed to give in­di­vid­u­als the right to con­trol their own in­for­ma­tion. The GDPR en­shrines a “right to era­sure,” also known as the “right to be for­got­ten,” as well as the right to trans­fer one’s per­sonal data among so­cial me­dia com­pa­nies, cloud stor­age providers, and oth­ers. The Euro­pean reg­u­la­tion also cre­ates new pro­tec­tions against al­go­rithms, in­clud­ing the “right to an ex­pla­na­tion” of de­ci­sions made through au­to­mated pro­cess­ing. So when a Euro­pean credit card is­suer de­nies an ap­pli­ca­tion, the ap­pli­cant will be able to learn the rea­son for the de­ci­sion and chal­lenge it. Cus­tomers can also in­voke a right to hu­man in­ter­ven­tion. Com­pa­nies found in vi­o­la­tion are sub­ject to fines ris­ing into the bil­lions of dol­lars. Reg­u­la­tion has been mov­ing in the op­po­site di­rec­tion in the United States, where no fed­eral leg­is­la­tion pro­tects per­sonal data. The Amer­i­can ap­proach is largely the honor sys­tem, sup­ple­mented by laws that pre­date the Internet, such as the Fair Credit Re­port­ing Act of 1970. In con­trast to Europe’s Data Pro­tec­tion Au­thor­i­ties, the US Fed­eral Trade Com­mis­sion has only min­i­mal au­thor­ity to as­sess civil penal­ties against com­pa­nies for pri­vacy vi­o­la­tions or data breaches. The Fed­eral Com­mu­ni­ca­tions Com­mis­sion (FCC) re­cently re­pealed its net neu­tral­ity rules, which were among the few pro­tec­tions re­lat­ing to dig­i­tal tech­nol­ogy.

These di­ver­gent ap­proaches, one reg­u­la­tory, the other dereg­u­la­tory, fol­low the same pat­tern as an­titrust en­force­ment, which faded in Wash­ing­ton and be­gan flour­ish­ing in Brus­sels dur­ing the Ge­orge W. Bush ad­min­is­tra­tion. But there is a con­vinc­ing case that when it comes to over­see­ing the use and abuse of al­go­rithms, nei­ther the Euro­pean nor the Amer­i­can ap­proach has much to of­fer. Au­to­mated de­ci­sion-mak­ing has rev­o­lu­tion­ized many sec­tors of the econ­omy and it brings real gains to so­ci­ety. It also threat­ens pri­vacy, au­ton­omy, demo­cratic prac­tice, and ideals of so­cial equality in ways we are only be­gin­ning to ap­pre­ci­ate.

At the sim­plest level, an al­go­rithm is a se­quence of steps for solv­ing a prob­lem. The in­struc­tions for us­ing a cof­feemaker are an al­go­rithm for con­vert­ing in­puts (grounds, fil­ter, wa­ter) into an output (cof­fee). When peo­ple say they’re wor­ried about the power of al­go­rithms, how­ever, they’re talk­ing about the ap­pli­ca­tion of so­phis­ti­cated, of­ten opaque, soft­ware pro­grams to enor­mous data sets. These pro­grams em­ploy ad­vanced sta­tis­ti­cal meth­ods and ma­chine-learn­ing tech­niques to pick out pat­terns and cor­re­la­tions, which they use to make pre­dic­tions. The most ad­vanced among them, in­clud­ing a sub­class of ma­chine-learn­ing al­go­rithms called “deep neu­ral net­works,” can in­fer com­plex, non­lin­ear re­la­tion­ships that they weren’t specif­i­cally pro­grammed to find.

Pre­dic­tive al­go­rithms are in­creas­ingly central to our lives. They de­ter­mine ev­ery­thing from what ads we see on the Internet, to whether we are flagged for in­creased se­cu­rity screen­ing at the air­port, to our med­i­cal di­ag­noses and credit scores. They lie be­hind two of the most pow­er­ful prod­ucts of the dig­i­tal in­for­ma­tion age: Google Search and Face­book’s News­feed. In many re­spects, ma­chine-learn­ing al­go­rithms are a boon to hu­man­ity; they can map epi­demics, re­duce en­ergy con­sump­tion, per­form speech recog­ni­tion, and pre­dict what shows you might like on Net­flix. In other re­spects, they are trou­bling. Face­book uses AI al­go­rithms to dis­cern the men­tal and emo­tional states of its users. While Mark Zucker­berg em­pha­sizes the ap­pli­ca­tion of this tech­nique to sui­cide pre­ven­tion, op­por­tu­ni­ties for op­ti­miz­ing ad­ver­tis­ing may pro­vide the stronger com­mer­cial in­cen­tive.

In many cases, even the de­vel­op­ers of al­go­rithms that em­ploy deep learn­ing tech­niques can­not fully ex­plain how they pro­duce their re­sults. The Ger­man startup SearchInk has pro­grammed a hand­writ­ing recog­ni­tion al­go­rithm that can pre­dict with 80 per­cent ac­cu­racy whether a sam­ple was penned by a man or wo­man. The data sci­en­tists who in­vented it do not know pre­cisely how it does this. The same is true of the much-crit­i­cized “gay faces” al­go­rithm, which can, ac­cord­ing to its Stan­ford Univer­sity cre­ators, dis­tin­guish the faces of ho­mo­sex­ual and het­ero­sex­ual men with 81 per­cent ac­cu­racy. They have only a hy­poth­e­sis about what cor­re­la­tions the al­go­rithm might be find­ing in pho­tos (nar­rower jaws and longer noses, pos­si­bly).

Ma­chine learn­ing of the kind used by the gay faces al­go­rithm is at the cen­ter of sev­eral ap­palling episodes. In 2015, the Google Pho­tos app la­beled pic­tures of black peo­ple as go­ril­las. In an­other in­stance, re­searchers found that kitchen ob­jects were associated with women in Mi­crosoft and Face­book im­age li­braries, while sport­ing equip­ment pre­dicted male­ness. Even a man stand­ing at a stove was la­beled a wo­man. In yet an­other case, Google searches for black-sound­ing first names like Trevon, Lak­isha, and Dar­nell were 25 per­cent more likely to re­turn ar­rest-re­lated ad­ver­tise­ments—in­clud­ing for web­sites that al­low you to check a per­son’s crim­i­nal record—than those for white-sound­ing names. Like dogs that bark at black peo­ple, ma­chine­learn­ing al­go­rithms lack the con­scious in­ten­tion to be racist, but seem some­how to ab­sorb the bias around them. They of­ten be­have in ways that re­flect pat­terns and prej­u­dices deeply em­bed­ded in his­tory and so­ci­ety.

The en­cod­ing of racial and gender dis­crim­i­na­tion via soft­ware de­sign re­li­ably in­spires an out­cry. In try­ing to fight bi­ased al­go­rithms, how­ever, we run into two re­lated prob­lems. The first is their im­pen­e­tra­bil­ity. Al­go­rithms have the le­gal pro­tec­tion of trade se­crets, and at Google and Face­book they are as closely guarded as the fa­bled recipe for Coca-Cola. But even mak­ing al­go­rithms trans­par­ent would not make them in­tel­li­gi­ble, as Frank Pasquale ar­gues in his in­flu­en­tial book The Black Box So­ci­ety: The Se­cret Al­go­rithms That Con­trol Money and In­for­ma­tion (2015). If the cre­ators of com­plex ma­chine-learn­ing sys­tems can­not ex­plain how they pro­duce their re­sults, ac­cess to the source code will at best give ex­perts a lim­ited abil­ity to dis­cover and ex­pose in­her­ent flaws. Mean­ing­less trans­parency threat­ens to lead to the same dead end as mean­ing­less con­sent, whereby end-user li­cense agree­ments (EULAs) filled with le­gal jar­gon fail to en­cour­age cau­tion or un­der­stand­ing.

The sec­ond prob­lem is dif­fused re­spon­si­bil­ity. Like the racist dog, al­go­rithms have been pro­grammed with in­puts that re­flect the as­sump­tions of a ma­jor­ity-white so­ci­ety. Thus they may am­plify bi­ases built into his­tor­i­cal data, even when pro­gram­mers at­tempt to ex­plic­itly ex­clude prej­u­di­cial vari­ables like race. If ma­chines are learn­ing on their own, hu­man ac­count­abil­ity be­comes trick­ier to as­cribe. We en­counter this eva­sion of re­spon­si­bil­ity nearly ev­ery time an al­go­rith­mic tech­nol­ogy comes un­der fire. In 2016, a ProPublica in­ves­ti­ga­tion re­vealed that Face­book’s ad­ver­tis­ing por­tal was al­low­ing land­lords to pre­vent African-Amer­i­cans, Lati­nos, and other “eth­nic affin­ity” groups from see­ing their ads, in ap­par­ent vi­o­la­tion of the Fair Hous­ing Act and other laws. Face­book blamed ad­ver­tis­ers for mis­us­ing its al­go­rithm and pro­posed a bet­ter ma­chine-learn­ing al­go­rithm as a so­lu­tion. The pre­dictable ten­dency at tech­nol­ogy com­pa­nies is to clas­sify moral fail­ings as tech­ni­cal is­sues and re­ject the need for di­rect hu­man over­sight. Face­book’s new tool was sup­posed to flag at­tempts to place dis­crim­i­na­tory ads and re­ject them. But when the same jour­nal­ists checked again a year later, Face­book was still ap­prov­ing the same kinds of bi­ased ads; it re­mained a sim­ple mat­ter to of­fer ren­tal hous­ing while ex­clud­ing such groups as African-Amer­i­cans, moth­ers of high school stu­dents, Span­ish speak­ers, and peo­ple in­ter­ested in wheel­chair ramps.1


The hope­ful as­sump­tion that soft­ware would be im­mune to the prej­u­dices of hu­man de­ci­sion-mak­ers has swiftly given way to the trou­bling re­al­iza­tion that os­ten­si­bly neu­tral tech­nolo­gies can re­flect and en­trench pre­ex­ist­ing bi­ases. Two new books ex­plore as­pects of this in­sid­i­ous po­ten­tial. In Al­go­rithms of Op­pres­sion, Safiya Umoja Noble, who teaches at the Univer­sity of South­ern Cal­i­for­nia’s An­nen­berg School of Com­mu­ni­ca­tion, pro­poses that “marginal­ized peo­ple are ex­po­nen­tially harmed by Google.”

This is an in­ter­est­ing hy­poth­e­sis, but Noble does not sup­port it. In­stead, she in­dicts Google with anti-im­pe­ri­al­ist rhetoric. The failed Google Glass project epit­o­mizes the company’s “neo­colo­nial tra­jec­to­ries.” Internet porn, avail­able via Google, is “an ex­pan­sion of ne­olib­eral cap­i­tal­ist in­ter­ests.” Google’s search dom­i­nance is a form of “cul­tural im­pe­ri­al­ism” that “only fur­ther en­trenches the prob­lem­atic iden­ti­ties in the me­dia for women of color.” Noble ex­em­pli­fies the trou­bling aca­demic ten­dency to view “free speech” and “free ex­pres­sion,” which she frames in quo­ta­tion marks, as tools of op­pres­sion. Her pre­ferred so­lu­tion, which she doesn’t ex­plore at any level of prac­ti­cal de­tail, is to cen­sor of­fen­sive web­sites or, as she puts it, “sus­pend the cir­cu­la­tion of racist and sex­ist ma­te­rial.” It’s hard

to imag­ine an answer to the prob­lem of al­go­rith­mic in­jus­tice that could be fur­ther off base. It might be tech­ni­cally if not legally pos­si­ble to block po­lit­i­cally sen­si­tive terms from Internet searches and to de­mand that search en­gines fil­ter their re­sults ac­cord­ingly. China does this with the help of a vast army of hu­man cen­sors. But even if throw­ing out the First Amend­ment doesn’t ap­pall you, it wouldn’t ac­tu­ally ad­dress the prob­lem of im­plicit bias. The point about al­go­rithms is that they can en­code and per­pet­u­ate dis­crim­i­na­tion un­in­ten­tion­ally with­out any con­scious ex­pres­sion ever tak­ing place.

Noble bases her cri­tique of Google pri­mar­ily on a sin­gle out­ra­geous ex­am­ple. As re­cently as 2011, if you searched for “black girls,” the first sev­eral re­sults were for pornog­ra­phy. (The same was true for “Asian girls” and “Latina girls” but not to the same ex­tent for “white girls.”) This is a gen­uine al­go­rith­mic hor­ror story, but Noble fails to ap­pre­ci­ate Google’s un­ex­pected re­sponse: it chose to re­place the pornog­ra­phy with so­cially con­struc­tive re­sults. If you search to­day for “black girls,” the first re­turn is for a non­profit called Black Girls Code that en­cour­ages African-Amer­i­can girls to pur­sue ca­reers in soft­ware engi­neer­ing. Pornog­ra­phy is blocked, even on the later pages (though it re­mains easy enough to find by sub­sti­tut­ing other search terms). By over­rid­ing the “or­ganic” re­sults, Google ac­knowl­edged the moral fail­ure of its pri­mary prod­uct. Pro­duc­ing a de­cent re­sult, it rec­og­nized, re­quired ex­actly what Face­book has re­sisted pro­vid­ing for its ad­ver­tis­ing por­tal: an in­ter­po­si­tion of hu­man de­cency. When it was a less ma­ture company, Google re­jected this kind of so­lu­tion too. In 2004, it re­fused de­mands to in­ter­vene in its re­sults when the search for “Jew” pro­duced as its top re­turn an anti-Semitic site called Jewwatch.com. To­day, Google rou­tinely down­grades what it calls of­fen­sive and mis­lead­ing search re­sults and au­to­com­plete sug­ges­tions, em­ploy­ing an army of search-qual­ity raters to ap­ply its guide­lines.2 This so­lu­tion points in a more promis­ing di­rec­tion: su­per­vi­sion by sen­sate hu­man be­ings rather than cat­e­gor­i­cal sup­pres­sion. It’s not clear whether it might sat­isfy Noble, who isn’t much in­ter­ested in dis­tinc­tions be­tween Google’s past and present prac­tices.

Vir­ginia Eubanks’s Au­tomat­ing Inequal­ity, which turns from the pri­vate sec­tor to the pub­lic sec­tor, gets much closer to the heart of the prob­lem. Its ar­gu­ment is that the use of au­to­mated de­ci­sion-mak­ing in so­cial service pro­grams cre­ates a “dig­i­tal poor­house” that per­pet­u­ates the kinds of neg­a­tive moral judg­ments that have al­ways been at­tached to poverty in Amer­ica. Eubanks, a po­lit­i­cal sci­en­tist at SUNY Al­bany, re­ports on three pro­grams that epit­o­mize this du­bi­ous in­no­va­tion: a wel­fare re­form ef­fort in In­di­ana, an al­go­rithm to dis­trib­ute scarce sub­si­dized apart­ments to home­less peo­ple in Los An­ge­les, and an­other de­signed to re­duce the risk of child en­dan­ger­ment in Al­legheny County, Penn­syl­va­nia. Former In­di­ana gov­er­nor Mitch Daniels’s dis­as­trous ef­fort to pri­va­tize and au­to­mate the process for de­ter­min­ing wel­fare el­i­gi­bil­ity in In­di­ana pro­vides one kind of sup­port for her the­sis. Run­ning for of­fice in 2004, Daniels blamed the state’s Fam­ily and So­cial Ser­vices Ad­min­is­tra­tion for en­cour­ag­ing wel­fare de­pen­dency. Af­ter IBM, in con­cert with a po­lit­i­cally con­nected lo­cal firm, won the bil­lion-dol­lar con­tract to au­to­mate the sys­tem, Daniels’s man­date to “re­duce in­el­i­gi­ble cases” and in­crease the speed of el­i­gi­bil­ity de­ter­mi­na­tions took prece­dence over help­ing the poor. The new sys­tem was clearly worse in a va­ri­ety of ways. An enor­mous back­log de­vel­oped, and er­ror rates sky­rock­eted. The newly dig­i­tized sys­tem lost its hu­man face; case­work­ers no longer had the fi­nal say in de­ter­min­ing el­i­gi­bil­ity. Re­cip­i­ents who couldn’t get through to the call cen­ter re­ceived “fail­ure to co­op­er­ate” no­tices. In the words of one state em­ployee, “The rules be­came brit­tle. If [ap­pli­cants] didn’t send some­thing in, one of thirty doc­u­ments, you sim­ply closed the case for fail­ure to com­ply . . . . You couldn’t go out of your way to help some­body.” Des­per­ately ill chil­dren were de­nied Med­i­caid cov­er­age. Be­tween 2006 and 2008, Eubanks writes, In­di­ana de­nied more than a mil­lion ap­pli­ca­tions for ben­e­fits, an in­crease of more than 50 per­cent, with a strongly dis­parate im­pact on black ben­e­fi­cia­ries. In 2000, African-Amer­i­cans made up 46.5 per­cent of the state’s re­cip­i­ents of TANF, the main fed­er­ally sup­ported wel­fare pro­gram. A decade later, they made up 32.1 per­cent. Things got so bad that Daniels even­tu­ally had to ac­knowl­edge that the ex­per­i­ment had failed and can­cel the con­tract with IBM.


To para­phrase Amos Tver­sky, the In­di­ana ex­per­i­ment may have less to say about ar­ti­fi­cial in­tel­li­gence than about nat­u­ral stu­pid­ity. The project didn’t de­ploy any so­phis­ti­cated tech­nol­ogy; it merely pro­vided tech­no­log­i­cal cover for an ef­fort to push peo­ple off wel­fare. By con­trast, the Al­legheny Fam­ily Screen­ing Tool (AFST)—an al­go­rithm de­signed to pre­dict ne­glect and abuse of chil­dren in the county that in­cludes Pitts­burgh—is a cut­ting-edge ma­chine-learn­ing al­go­rithm de­vel­oped by a team of econ­o­mists at the Auck­land Univer­sity of Tech­nol­ogy. Tak­ing in such vari­ables as a par­ent’s wel­fare sta­tus, men­tal health, and crim­i­nal jus­tice record, the AFST pro­duces a score that is meant to pre­dict a child’s risk of en­dan­ger­ment.

Pub­lic re­sis­tance pre­vented the launch of an ob­ser­va­tional ex­per­i­ment with the same al­go­rithm in New Zealand. But in Al­legheny County, a well-liked and data-minded di­rec­tor of the Depart­ment of Hu­man Ser­vices saw it as a way to max­i­mize the ef­fi­ciency of di­min­ish­ing re­sources pro­vided by the Penn­syl­va­nia as­sem­bly for child wel­fare pro­grams. The al­go­rithm was de­ployed there in 2016.

Eubanks, who spent time at the call cen­ter where re­ports are pro­cessed, ex­plains how the risk as­sess­ment works. When a call comes in to the child ne­glect and abuse hot­line, the al­go­rithm mines stored data and fac­tors in other vari­ables to rate the risk of harm on a scale of zero to twenty. She ob­serves that its pre­dic­tions of­ten defy com­mon sense: “A 14-year-old liv­ing in a cold and dirty house gets a risk score al­most three times as high as a 6-year-old whose mother sus­pects he may have been abused and who may now be home­less.” And in­deed, the al­go­rithm is of­ten wrong, pre­dict­ing high lev­els of ne­glect and abuse that are not sub­stan­ti­ated in fol­low-up in­ves­ti­ga­tions by case­work­ers, and fail­ing to pre­dict much ne­glect and abuse that is found in sub­se­quent months. In the­ory, hu­man screen­ers are sup­posed to use the AFST as sup­port for their de­ci­sions, not as the de­ci­sion maker. “And yet, in prac­tice, the al­go­rithm seems to be train­ing the in­take work­ers,” she writes. Hu­mans tend to de­fer to high scores pro­duced by con­cep­tu­ally flawed soft­ware. Among the AFST’s flaws is that it pre­dicts harm to black and bira­cial chil­dren far more of­ten than to white ones. How does racial bias come through in the al­go­rithm? To a large ex­tent, the AFST is sim­ply mir­ror­ing un­fair­ness built into the old hu­man sys­tem. Fortyeight per­cent of the chil­dren in fos­ter care in Al­legheny County are African-Amer­i­can, even though only 18 per­cent of the to­tal pop­u­la­tion of chil­dren are African-Amer­i­can. By us­ing call re­fer­rals as its chief “out­come vari­able,” the al­go­rithm per­pet­u­ates this dis­par­ity. Peo­ple call the hot­line more of­ten about black and bira­cial fam­i­lies than about white ones. This re­flects in part an ur­ban/ru­ral racial di­vide. Since anony­mous com­plaints by neigh­bors lead to a higher score, the sys­tem ac­cen­tu­ates a pre­ex­ist­ing bias to­ward notic­ing ur­ban fam­i­lies and ig­nor­ing ru­ral ones. Ad­di­tional un­fair­ness comes from the data ware­house, which stores ex­ten­sive in­for­ma­tion about peo­ple who use a range of so­cial ser­vices, but none for fam­i­lies that don’t. “The pro­fes­sional mid­dle class will not stand for such in­tru­sive data gath­er­ing,” Eubanks writes. Poor peo­ple learn two con­tra­dic­tory lessons from be­ing as­signed AFST scores. One is the need to act def­er­en­tially around case­work­ers for fear of hav­ing their chil­dren taken away and placed in fos­ter care. The other is to try to avoid ac­cess­ing so­cial ser­vices in the first place, since do­ing so brings more sus­pi­cion and sur­veil­lance. Eubanks talks to some of the ap­par­ent vic­tims of this sys­tem: work­ing­class par­ents do­ing their best who are con­stantly surveilled, in­ves­ti­gated, and su­per­vised by state au­thor­i­ties, of­ten as a re­sult of what are ap­par­ently vendetta calls to the hot­line from land­lords, exspouses, or neigh­bors dis­turbed by noisy par­ties. “Or­di­nary be­hav­iors that might raise no eye­brows be­fore a high AFST score be­come con­fir­ma­tion for the de­ci­sion to screen them in for in­ves­ti­ga­tion,” she writes. “A par­ent is now more likely to be re-re­ferred to a hot­line be­cause the neigh­bors saw child pro­tec­tive ser­vices at her door last week.” What emerges is a vi­cious cy­cle in which the self-val­i­dat­ing al­go­rithm pro­duces the be­hav­ior it pre­dicts, and pre­dicts the be­hav­ior it pro­duces.

This kind of feed­back loop helps to ex­plain the “racist dog” phe­nom­e­non of os­ten­si­bly race-neu­tral crim­i­nal jus­tice al­go­rithms. If a cor­re­la­tion of dark skin and crim­i­nal­ity is re­flected in data based on pat­terns of racial pro­fil­ing, then pro­cess­ing his­tor­i­cal data will pre­dict that blacks will com­mit more crimes, even if nei­ther race nor a proxy for race is en­coded as an in­put vari­able. The pre­dic­tion brings more su­per­vi­sion, which sup­ports the pre­dic­tion. This seems to be what is go­ing on with the Cor­rec­tional Of­fender Man­age­ment Pro­fil­ing for Al­ter­na­tive Sanc­tions tool (COMPAS). ProPublica stud­ied this risk-as­sess­ment al­go­rithm, which judges around the coun­try use to help make de­ci­sions about bail, sen­tenc­ing, and pa­role, as part of its in­valu­able “Ma­chine Bias” se­ries.3

COMPAS pro­duces nearly twice the rate of false pos­i­tives for blacks that it does for whites. In other words, it is much more likely to in­ac­cu­rately pre­dict that an African-Amer­i­can de­fen­dant will com­mit a sub­se­quent vi­o­lent of­fense than it is for a white de­fen­dant. Be­ing un­em­ployed or hav­ing a par­ent who went to prison raises a pris­oner’s score, which can bring higher bail, a longer prison sen­tence, or de­nial of pa­role. Cor­re­la­tions re­flected in his­tor­i­cal data be­come in­vis­i­bly en­trenched in pol­icy with­out pro­gram­mers hav­ing ill in­ten­tions. Quan­ti­fied in­for­ma­tion nat­u­rally points back­ward. As Cathy O’Neil puts it in Weapons of Math De­struc­tion: “Big Data pro­cesses cod­ify the past. They do not in­vent the fu­ture.”4 In the­ory, the EU’s “right to an ex­pla­na­tion” pro­vides a way to at least see and un­der­stand this kind of em­bed­ded dis­crim­i­na­tion. If we had some­thing re­sem­bling the Gen­eral Data Pro­tec­tion Reg­u­la­tion in the United States, a pro­pri­etary al­go­rithm like COMPAS could not hide be­hind com­mer­cial se­crecy. De­fen­dants could the­o­ret­i­cally sue to un­der­stand how they were scored.

But se­crecy isn’t the main is­sue. The AFST is a fully trans­par­ent al­go­rithm that has been sub­ject to ex­ten­sive dis­cus­sion in both New Zealand and the United States. While this means that flaws can be dis­cov­ered, as they have been by Eubanks, the re­sult­ing knowl­edge is prob­a­bly too es­o­teric and tech­ni­cal to be of much use. Pub­lic pol­icy that hinges on un­der­stand­ing the dis­tinc­tions among out­come vari­ables, pre­dic­tion vari­ables, train­ing data, and val­i­da­tion data seems cer­tain to be­come the do­main of tech­nocrats. An ex­pla­na­tion is not what’s wanted.

What’s wanted is for the harm not to have oc­curred in the first place, and not to con­tinue in the fu­ture.

Fol­low­ing O’Neil, Eubanks pro­poses a Hip­po­cratic oath for data sci­en­tists, whereby they would vow to re­spect all peo­ple and to not com­pound pat­terns of dis­crim­i­na­tion. Whether this would have much ef­fect or be­come yet an­other form of mean­ing­less con­sent is open to de­bate. But she is cor­rect that the answer must come in the form of eth­i­cal com­punc­tion rather than more data and bet­ter min­ing of it. Sys­tems sim­i­lar to the AFST and COMPAS con­tinue to be im­ple­mented in ju­ris­dic­tions across the coun­try. Al­go­rithms are de­vel­op­ing their ca­pa­bil­i­ties to reg­u­late hu­mans faster than hu­mans are fig­ur­ing out how to reg­u­late al­go­rithms.

An il­lus­tra­tion show­ing fa­cial land­marks ex­tracted with widely used fa­cial recog­ni­tion al­go­rithms; from a re­cent study by Stan­ford re­searchers Michal Kosin­ski and Yilun Wang show­ing that such al­go­rithms can re­veal sex­ual ori­en­ta­tion

Newspapers in English

Newspapers from USA

© PressReader. All rights reserved.