Automated Justice: A Review of Weapons of Math Destruction

Stephen Raher reviews Cathy O’Neil's book, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy.

by Stephen Raher, April 18, 2017

Cathy O’Neil has a Ph.D in math, and has worked in academia, finance, and data analytics. Fortunately for everyone, she recently decided to use her skills to write an informative and accessible exploration of “Big Data” in her book Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy.

Referencing the collateral damage caused by weapons of mass destruction, O’Neil coined the term “weapons of math destruction” (WMDs). She defines WMDs as opaque mathematical models that embed human prejudice, misunderstanding, and bias into the software systems that automate numerous aspects of our lives. Her book covers several types of these models and the frustrating injustices they can perpetrate. In addition to case studies about credit scoring, online advertising, employment, and insurance, O’Neil discusses the use of WMDs in the criminal justice system. In particular, the book considers models used to deploy police, predict a defendant’s chances of recidivism, and calculate prison sentences.

Weapons of Math Destruction covers a lot of ground, but four distinct themes emerge in connection with the criminal justice system’s use of WMDs.

1) Fairness

The book repeatedly points to the unfairness that results from WMDs. O’Neil gives examples of algorithms that map crime so that police can predict crime hotspots, as well as models that take demographic information about a criminal defendant and calculate a “risk score” purporting to predict the likelihood that the defendant will recidivate. Although the creators of such WMDs usually claim not to consider race as a factor, facially neutral data like neighborhood can act as a highly accurate proxy for race. As a result, mathematical models, which are marketed as supposedly removing human bias from the system, often perpetuate racial discrimination.

WMDs also work to the disadvantage of low-income people. Crime-mapping programs direct police to focus on poor neighborhoods with high crime rates, leading to increased arrests in those areas (usually for “quality of life” non-violent offenses), thus reinforcing the initial data that led to heavier policing of poor neighborhoods. Residents from these localities who are convicted of crimes are—because their friends and neighbors are more likely to have criminal records—more likely to be flagged by risk-scoring models as high recidivism risks and receive longer sentences. As other researchers have noted, this faulty logic shifts responsibility for community-wide problems to individuals who have no control over neighborhood conditions like racial profiling, inadequate job opportunities, limited educational opportunities, and lack of access to mental health care. In O’Neil’s words, because WMDs have no way of considering fairness, “the result is massive, industrial production of unfairness. If you think of a WMD as a factory, unfairness is the black stuff belching out of the smoke stacks. It’s an emission, a toxic one.”

It’s also important to remember that unfairness isn’t always an unintended byproduct of WMDs. After publishing Weapons of Math Destruction, O’Neil has written about the current presidential administration’s intentional steps to distort data in an effort to criminalize immigrants.

2) Legality

In addition to the broader issues of fairness, O’Neil points to narrower questions of legality. Even though recidivism risk scores are problematic from the outset (as discussed above), they could arguably be useful as one of several factors in making certain decisions (for example, identifying people who could benefit from intensive education or therapy programs). But grave legal issues are implicated when these models are used in determining a criminal sentence (a practice that is currently used in nine states, according to a 2016 ProPublica report, and which seems to be gaining in popularity). The U.S. Constitution guarantees that a criminal defendant can confront witnesses and challenge the evidence presented against him or her. But O’Neil explains that recidivism risk scores, unlike witness testimony, cannot be recorded and challenged in court, but instead “are tucked away in algorithms, intelligible only to a tiny elite.” Moreover, risk scores can be based on information that would not be admissible in a court proceeding, such as the criminal background of a defendant’s friends and family members, or the crime rate in his or her neighborhood.

3) Garbage in, garbage out

Ever since the early days of computer programming, programmers have acknowledged that inaccurate input data will produce inaccurate results. In the criminal justice context, this arises when deciding what type of crime statistics to feed into a WMD. When discussing crime-mapping, O’Neil posits that if a model is built on data about burglary, car theft, and violent crime, then perhaps the results could be useful in deploying limited police resources (although, even then, she acknowledges that crime reports and arrest records aren’t particularly reliable proxies for the true amount and nature of criminal activity). The big problem comes when nuisance offenses like vagrancy, panhandling, and simple drug possession are fed into models. These data distort crime calculations, fueling the proliferation of “broken windows policing” and ever-growing numbers of arrests.

On the flip side of the coin, plenty of harmful crime is not incorporated into WMDs. Financial firms defraud customers (sometimes nearly bringing the global economy to a halt) and industries violate environmental laws—but these harmful activities are not included in crime-prediction models, and the algorithms do not dispatch battalions of police to look for violations in wealthy suburbs and gated communities. As O’Neil writes, “[t]he result is that we criminalize poverty, believing all the while that our tools are not only scientific but fair.”

4) Using data for good

Finally, O’Neil expresses her frustration with the fact that sophisticated mathematical models can be used to improve society, but such deployment is not common because it would threaten entrenched powers. For example, she notes that prisons collect massive amounts of data on incarcerated persons, yet this information is not used to tackle questions like: what are the impacts of solitary confinement, what are effective tools for combatting sexual assault in prison, and what types of prison experiences (big or small) effectively reduce recidivism. Data that could be used to address issues like these are either not held in a usable format, or are purposely withheld from researchers. According to O’Neil, prison administrators “use data to justify the workings of the system but not to question or improve the system.”

Food for thought

I found Weapons of Math Destruction to be a good read because it provides an expert’s insight into how WMDs impact all of us. But this is a huge topic, and at roughly 200 pages, O’Neil’s book can’t cover all the ground, particularly in regards to criminal justice. There are many topics that remain lurking in the background, including three that come immediately to my mind.

First is the proliferation of data sources that can potentially feed WMDs. When discussing criminal-justice WMDs, O’Neil frequently mentions data comprised of arrest records, court records, and other public documents, but doesn’t explore the growth of “alternative” datasets that could be sucked up into risk-scoring algorithms. There is cause to worry on this front. In a report that I wrote for the Prison Policy Initiative last year, I noted that some prisons are embracing electronic messaging as an alternative to mailed correspondence to and from incarcerated persons. These services are typically operated by private contractors that provide users with little or no privacy protections. This raises the all-to-real possibility that years’ worth of correspondence between an incarcerated person and their family could be used as data in a risk-scoring WMD. As another example, the federal Bureau of Prisons is trying to weaken financial privacy protections so that it can collect banking information on people who send money to people in prison (the Prison Policy Initiative and other groups have formally opposed this move, and BOP has not yet finalized the draft rule).

Second, the ease with which data travels can mean that it’s harder for formerly-incarcerated people to escape the stigma attached to a criminal record. As more businesses and government agencies rely on WMDs to automate decision-making, criminal records will inevitably flow through more databases, thus ensuring an ever widening net that prevents formerly incarcerated people from meaningfully participating in economic, civic, and social life. Communities must start discussing fair ways to honor public access to information without condemning formerly incarcerated people to a permanent diminished tier of citizenship.

Finally, there is the fact that opaque and unfair WMDs developed in the criminal justice system today are likely to be applied to society-at-large tomorrow. News reports recently indicated that about half of the adult population of the U.S. is unwittingly contained in an FBI facial-recognition database that has a reported 15% inaccuracy rate (and is far more likely to misidentify Black people). While we can expect to see more expansion of such systems, the real question is where this leads: will experiments such as the FBI’s catalyze a thoughtful discussion of privacy, fairness, and expectations about risk; or, will they hasten a Minority Report-style society where people are punished because they might commit a crime in the future?