Algorithm gon’ get you
NEW BOOK EXPLORES HOW WE’RE TRAPPED BY BIG DATA
Reading Cathy O’Neil’s indispensable Weapons of Math Destruction, her thorough and often upsetting account of the complex predictive and sorting algorithms that have become powerful forces in the everyday lives of most people, a line from the late great Ursula Franklin’s The Real World of Technology kept echoing in my mind. In 1990, Franklin wrote, “We have lost the institution of government in terms of responsibility and accountability to the people. We now have nothing but a bunch of managers, who run the country to make it safe for technology.” O’Neil’s book in places reads like a confirmation of Franklin’s, as she describes the minute ways in which certain algorithms, deployed in realms of human endeavour from finance to job hunting to criminal justice, are not only defining and interpreting the worlds into which they’ve been unleashed, but are also actively shaping them.
O’Neil’s career as a Wall Street quant and data scientist gives her an insider’s view into the rise of Big Data, the catch-all term for crunching huge amounts of data pertaining to hundreds or thousands or millions of people at once, looking for emergent trends that help predict behaviour. Despite the technical complexity of its subject, Weapons of Math Destruction lucidly guides readers through these complex modelling systems that O’Neil describes as “opinions embedded in mathematics.” Not all such models attract her ire: the titular “weapons of math destruction” are a specific species of algorithmic bad actors in which “poisonous assumptions are camouflaged by math and go largely untested and unquestioned.” These “WMDs” tend to target the poor or marginalized, contain self- validating feedback loops, and can have sharply negative impacts on those caught in their nets.
O’Neil highlights the centrality of the profit motive in the development and deployment of these algorithms, which are not, for the most part, deployed to make the lives of those whom they sort easier or better. These algorithms execute a series of invisible transformations: person into consumer, consumer into mark, mark into dollar sign. Profitability becomes not only the stamp of a successful algorithm, O’Neil notes, but also “a stand- in or proxy for the truth.” Individuals are elided into data points, and it becomes easy to confuse the algorithm’s descriptive correlations for those things they are attempting to describe.
One prominent example of this is the use of students’ standardized test scores as a proxy for school quality, as occurred in the United States under a set of regulations called No Child Left Behind. NCLB valorized standardized testing as the key metric for evaluating teachers and schools, leading public schools in the States to devote hours to “teaching to the test,” trying to push up test scores to avoid firings, funding cuts, or even school closures. Scrambles to avoid falling behind in test score rankings have led to at least one major cheating scandal, where teachers in an atrisk school in Atlanta, Ga., changed students’ answers on a statewide standardized exam to keep their school from being shuttered.
These algorithms hit the most precarious in our societies the hardest by explicitly targeting them for malicious services (such as payday loans or for-profit college degrees) or by silently, invisibly removing them from contention for jobs, apartments, a place at university, or even by lining them up for harsher penalties from the criminal justice system. O’Neil describes the impact of a mathematical model meant to gauge criminal recidivism risk this way:
“A person who scores ‘ high risk’ is likely to be unemployed and to come from a neighbourhood where many of his friends and family have had run- ins with the law. Thanks in part to the resulting high score on the evaluation, he gets a longer prison sentence, locking him away for more years in a prison where he’s surrounded by fellow criminals — which raises the likelihood that he’ll return to prison. He is finally released into the same poor neighbourhood, this time with a criminal record, which makes it much harder to find a job. If he commits another crime, the recidivism model can claim another success. But in fact the model contributes to a toxic cycle and helps to sustain it.”
Though not all the malignant algorithms O’Neil examines are explicitly designed to separate a sucker from his money, it’s astonishing how often algorithms use money — whether one has it or doesn’t, or how much one is willing to spend to get out of an algorithm’s grasp — as the defining characteristic of a person. Often this takes the form of a credit score. In Canada and in the United States, it’s common for landlords to perform credit checks on prospective t enants, and some employers treat the automatic, algorithmic checking of an applicant’s credit as on par with checking their references, particularly with low- skill, entrylevel, high- churn jobs that are often lifelines for economically precarious populations. It’s distressing that access to basic human needs like housing or the means to earn a living is potentially dependent upon the shibboleth of credit-worthiness.
As Ursula Franklin noted, these algorithms have a shaping impact on the world around us and our behaviour in it. As algorithms are increasingly used as arbiters of quality, corporations and individuals are increasingly playing to the algorithms themselves, creating an increasingly tunnel- visioned game of maximizing algorithmic gain while disregarding costs and risks.
O’Neil describes the impact of U. S. News & World Report’s ambitious college ranking project, begun in 1983 in an attempt to remedy flagging newsstand sales, on schools themselves. As the U. S. News rankings became more popular among parents attempting to wade through the thousand of higher education options, colleges and universities began attempting to game specific factors in the U. S. News ranking algorithm (which was developed by journalists, not higher education experts), hoping to up their position. Suddenly colleges were paying admitted students to retake the SATs in order to increase the average score for the incoming class, or shifting their admissions and financial aid policies to appear more selective and increase their four-year graduation rate.
Did these moves make t hese colleges “better”? Maybe. But unless “better” can be measured algorithmically, that question is unlikely to be seriously asked or seriously answered. An over-reliance on these mathematical models prevents us from asking whether the quantification of knowledge, the endless translation of human concepts of quality and of worth into strings of numbers and collapsed data points is a good thing. Is quantitative knowledge always the best type knowledge for a given situation? While Weapons of Math Destruction does not dive into epistemology, it will hopefully open a space where such conversations can occur.
O’Neil’s book is an excellent primer on the ethical and moral risks of Big Data and an algorithmically dependent world. It compellingly describes algorithms ( and those who use them) behaving badly, and advocates for society to do better. O’Neil is no quantitative apostate. She believes in the beneficial potential of Big Data, and that it can do better. For those curious about how Big Data can help them and their businesses, or how it has been reshaping the world around them, Weapons of Math Destruction in an essential starting place.