HRDAG’s analysis and expertise continues to deepen the national conversation about police violence and criminal justice reform in the United States. In 2015 we began by considering undocumented victims of police violence, relying on the same methodological approach we’ve tested internationally for decades. Shortly after, we examined “predictive policing” software, and demonstrated the ways that racial bias is baked into the algorithms. Following our partners’ lead, we next considered the impact of bail, and found that setting bail increases the likelihood of a defendant being found guilty. We then broadened our investigations to examine the risk assessment tools that judges use to make decisions about pre-trial supervision, and we found evidence of racial bias in the tools. Most recently we have returned to considering the challenges of documenting police violence.
Violence committed by police officers in the United States is paradoxically highly documented and incompletely documented. We frequently encounter this situation in our work—some victims’ stories are told by many sources, while other victims’ stories are rarely if ever told. This is the gap statistics can fill.
In 2015, the US Bureau of Justice Statistics concluded that two federal efforts to document deaths that occur during arrest were woefully incomplete. Their analysis relied on the same statistical methods HRDAG’s team has been using internationally for decades, so we examined their approach, considering various assumptions required by the method. This work resulted in an essay in Granta magazine, “Violence in Blue,” written by our director of research Patrick Ball, which concluded that one-third of all Americans killed by strangers are killed by police. Following the murder of George Floyd by Minneapolis police officers on May, 2020, Patrick refreshed the article with a new introduction, writing, “This cannot be acceptable in democracy.”
Current work includes ongoing collaboration with several US-based partners. We are working with the Invisible Institute’s Citizens Police Data Project to design and maintain a data pipeline, to systematically process large quantities of documents describing potential police misconduct in Chicago.
Key publications on police violence:
Patrick Ball (2020). Refresh of Violence in Blue (Granta, March 2016) with new introduction. 8 June, 2020. © 2020 Granta.
Kristian Lum and Patrick Ball (2015). Estimating Undocumented Homicides with Two Lists and List Dependence. HRDAG, April 2015.
Kristian Lum and Patrick Ball (2015). How many police homicides in the US? A reconsideration. HRDAG, April 2015.Kristian Lum and Patrick Ball (2015). BJS Report on Arrest-Related Deaths: True Number Likely Much Greater. HRDAG, March 2015.
photo by Flickr user John Liu, CC-BY-2.0, modified by David Peters
The Problem with Predictive Policing
Among many law enforcement communities, predictive policing software tools such as PredPol are being adopted in an attempt to increase policing effectiveness and to eliminate bias among their agents. The claim made by the enforcement agencies and the software developers is that because predictive policing software uses data—and not human judgment—the tools are free of racial bias.
But our research shows that the use of “big data” reinforces racial bias in these tools, because the tools use data generated by policing practices that target people of color and the communities they live in .
Key publications on predictive policing:
William Isaac and Kristian Lum (2018). Setting the Record Straight on Predictive Policing and Race. In Justice Today. 3 January 2018. © 2018 In Justice Today / Medium.
Laurel Eckhouse (2017). Big data may be reinforcing racial bias in the criminal justice system. Washington Post. 10 February 2017. © 2017 Washington Post.
William Isaac and Kristian Lum (2016). Predictive policing violates more than it protects. USA Today. December 2, 2016. © USA Today.
Investigating Risk Assessment
In many judicial processes, judges rely on pre-trial risk assessment software tools to help them decide the fates of arrested persons. Will she be released on her own recognisance until the trial? Will she be given the option to pay bail in order to secure her release before the trial? Or will she be denied bail (and release)? The software programs that give judges a risk assessment rating are touted as “bias-free,” because they are algorithms—but our research shows that the data used to “train” the algorithms encode racial bias into the tools.
In examining different risk assessment tools used in different cities, we’ve collaborated on projects with San Francisco’s District Attorney, Chesa Boudin, and with the Criminal Justice Agency in New York City. We’ve also collaborated with The Safety + Justice Challenge to produce a primer on pre-trial risk assessment tools for judges, prosecutors, and defense attorneys.
Key publications on risk assessment:
Kristian Lum, Chesa Boudin and Megan Price (2020). The impact of overbooking on a pre-trial risk assessment tool. FAT* ’20: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. January 2020. Pages 482–491. https://doi.org/10.1145/3351095.3372846 ©ACM, Inc., 2020.
Kristian Lum and Tarak Shah (2019). Measures of Fairness for New York City’s Supervised Release Risk Assessment Tool. Human Rights Data Analysis Group. 1 October 2019. © HRDAG 2019.
Sarah L. Desmarais and Evan M. Lowder (2019). Pretrial Risk Assessment Tools: A Primer for Judges, Prosecutors, and Defense Attorneys. Safety and Justice Challenge, February 2019. © 2019 Safety and Justice Challenge. <<HRDAG’s Kristian Lum and Tarak Shah served as Project Members and made significant contributions to the primer.>>
Laurel Eckhouse, Kristian Lum, Cynthia Conti-Cook and Julie Ciccolini (2018). Layers of Bias: A Unified Approach for Understanding Problems With Risk Assessment. Criminal Justice and Behavior. November 23, 2018. © 2018 Sage Journals. All rights reserved. https://doi.org/10.1177/0093854818811379Kristian Lum (2017). Limitations of mitigating judicial bias with machine learning. Nature. 26 June 2017. © 2017 Macmillan Publishers Limited. All rights reserved. DOI 10.1038/s41562-017-0141.
Examining the Impact of Bail
In collaboration with partners at
In collaboration with the New York Legal Aid Society (NYLAS) and Stanford University assistant professor Mike Baiocchi, HRDAG has examined the effect of setting bail on a defendant’s likelihood of a guilty finding, either through a jury’s determination at trial or by taking a guilty plea in advance of trial. Spearheading this collaboration for HRDAG, lead statistician Kristian Lum has used NYLAS datasets and found, with our partners, that setting bail increases defendants’ likelihood of a guilty finding, usually through a plea deal, compared to the outcomes of defendants who are released or remanded.
Key publication on impact of bail:
Kristian Lum, Erwin Ma and Mike Baiocchi (2017). The causal impact of bail on case outcomes for indigent defendants in New York City. Observational Studies 3 (2017) 39-64. 31 October 2017. © 2017 Institute of Mathematical Statistics.
photo by Flickr user Daniel Schwen, CC-BY-4.0, modified by David Peters
Podcast: Risk Assessment Biases | Stats + Stories, Episode 147 | Tarak Shah | 2020 (28 minutes)
Podcast: Lifelong curiosity and Criminal Justice Reform through data | Origins, Episode 19 | Kristian Lum | 2020 (52 minutes)
Video: Predictive policing and machine learning | Skoll Foundation | Megan Price | 2018 (11 minutes)
Video: Predictive Policing | Data Society and Research Institute | Kristian Lum | 2016 (56 minutes)
Video: Tyranny of the Algorithm? Predictive Analytics and Human Rights | NYU School of Law | Patrick Ball and Kristian Lum | 2016 (51 minutes)