How many police homicides in the US? A reconsideration

(This post is co-authored by Patrick Ball and Kristian Lum.)

In early March, the Bureau of Justice Statistics published a report that estimated that in the period 2003-2009 and 2011, there were approximately 7427 homicides committed by police in the US. We responded that the method the analysts used, capture-recapture with two databases, is vulnerable to underestimation if the databases exhibit positive dependence. We conduct a thorough sensitivity analysis on the original independence model as applied to the police homicides databases. We used information from several other countries where our partners created multiple databases of homicides. We then used those data to estimate a range of plausible list dependence values. If the dependence between the two databases used to estimate the number of police homicides is similar to that seen in several other, contextually similar examples,  we estimate that the total number of victims of homicide by the police during 2003-2009 and 2011 is probably approximately 10,000. We have written a very detailed analysis of why we think this is true, available here. Here’s a shorter version:

In the approach used by the BJS analysts, if you work with only two databases, you must assume that the appearance of a homicide in one database makes that homicide no more or less likely to appear in the other. This assumption is called list independence, and it’s usually not true. In fact, one of the authors of the BJS report agreed that the two databases they used — the ARD and the SHR — are probably positively correlated. See the comments by Lance Couzens in this interview.

When we work with three or more databases, we can estimate the dependence among any pair of databases, thereby avoiding the independence assumption. We’ve done this kind of estimate in many projects, including Kosovo, Colombia, Guatemala, Sierra Leone (unpublished, but based on projects described here), and Syria (the estimates are unpublished, but the data are reviewed in reports made available here).

We have found that lists tend to be positively correlated. They may draw from some of the same underlying source documents; they may draw data from a population that tends to trust both documentation projects; or they may both tend to more frequently report the killings of prominent people while tending to ignore the killings of victims who are poorer, more rural, or from disadvantaged minority groups. Any of these conditions could create positive dependence among lists.

We decided to estimate the list dependence among all the pairs of lists in countries we’ve worked in before. As shown in Figure 1 (above), the list dependence is positive for most of the pairs of lists for most of the countries we’ve looked at. The next question about police homicides in the US is therefore, “What would the estimated total number of police homicides be, given the data used in the BJS report, if list dependence followed the patterns found in the other countries we’ve studied?”

Our report explains the math. Basically, for each of the projects with more than two lists that we mentioned above, we estimate the correlation between each pair of lists. We then apply each estimated correlation to the BJS data to obtain an  adjusted estimate of the number of police homicides. The adjustment assumes that the the correlation between the two lists used by the BJS is the same as the correlation we estimated on the given pair of lists from one of our previous projects.  Figure 2 provides the ranges of adjusted estimates. These estimates can be understood as a kind of sensitivity analysis of the BJS estimate to the likely list dependence.

Both Figures 1 and 2 are boxplots, graphs designed to show a range of values. The box shows the range from the 25th to 75th percentile of each distribution, the whiskers on each side show the approximate range of values. So, for example, if the true list dependence between the SHR and the ARD falls between the 25th to 75th percentile of list dependencies in Colombia, the range of estimated police homicides in the US is [7931, 13965], with the median at 10091.

We think the estimates of list dependence in Colombia are the most similar to the lists compiled by the US Bureau of Justice Statistics (the Arrest Related Deaths, ARD) and the FBI’s Supplementary Homicide Reports (SHR).  In our work on Colombia (see our report on the state of Casanare here), we include lists of homicides compiled by the Colombian government that are similar to the SHR. For example, we include a list from the Colombian National Police, and another from the Fiscalía General de la Nación (the Colombian ministry of justice), and another from the Colombian Commission of Jurists (an NGO that compiles reports by other NGOs and from the media). These lists seem to us pretty close to what the BJS team had to work with.

This analysis highlights two important facts: (1) the number of unrecorded police homicides in the US is large as a proportion of total homicides and (2) there remains a high degree of uncertainty as to the true number of police homicides. Despite our best efforts to use statistics to compensate for the fact that many police homicides remain undocumented, uncertainty as to the true number of homicides will always remain high without better and more complete reporting. We have worked in more than thirty countries experiencing violent conflict, and our experience teaches us that some fraction of homicides are always hidden. Nonetheless, government authorities should strive to document every police homicide. A more complete registry would be a huge step towards increasing our certainty about the true number.  HRDAG encourages the US authorities to record all homicides committed by police, and to make the lists of victims publicly available. Documenting deaths is a fundamental obligation of governments, and deaths resulting from government actions are perhaps the most important category of deaths to be recorded.

However, even in the most rigorous documentation contexts, some homicides are likely be unreported. All lists of homicide victims should be tested for completeness using methods like those used in the BJS report. We welcome the BJS report and estimates  as a critical step forward and congratulate the authors on bringing attention to under-documentation of police homicides in the US.

 CC BY-NC-SA, excluding images.

---