Clustering and Solving the Right Problem

In our database deduplication work, we’re trying to figure out which records refer to the same person, and which other records refer to different people. We write software that looks at tens of millions of pairs of records. We calculate a model that assigns each pair of records a probability that the pair of records refers to the same person. This step is called pairwise classification. However, there may be more than just one pair of records that refer to the same person. Sometimes three, four, or more reports of the same death are recorded. So once we have all ...

Read More


HRDAG and AHPN Launch Book Detailing Collaboration

Earlier this month, HRDAG and the Historic Archive of the National Police (AHPN) of Guatemala launched a book that represents a long-time collaboration between the two organizations. The book, “Una mirada al AHPN a partir de un studio de cuantitativo,” is, as the title states, a look at the Archive’s datasets via a quantitative study. Book authors are HRDAG executive director Megan Price and AHPN colleague Carolina López, with translations by Beatriz Vejarano. The book is available in Spanish and forthcoming in English. The book explains how HRDAG and the Archive ...

Read More


Megan Price Elected Board Member of Tor Project

Today The Tor Project announced that it has elected a new Board of Directors, and among them is HRDAG executive director Megan Price. The Tor Project is a nonprofit advocacy group that promotes online privacy and provides software that helps users opt out of online tracking. Megan and Patrick have long maintained that encryption and privacy are essential for enabling human rights work. Patrick's ideas are described in Monday's FedScoop story about encryption, human rights, and the U.S. State Department. “Human rights groups depend on strong cryptography in order ...

Read More


The task is a quantum of workflow

This post describes how we organize our work over ten years, twenty analysts, dozens of countries, and hundreds of projects: we start with a task. A task is a single chunk of work, a quantum of workflow. Each task is self-contained and self-documenting; I'll talk about these ideas at length below. We try to keep each task as small as possible, which makes it easy to understand what the task is doing, and how to test whether the results are correct. In the example I'll describe here, I'm going to describe work from our Syria database matching project, which ...

Read More


Momentous Verdict against Hissène Habré

Today we’re very pleased to hear of the verdict finding Hissène Habré guilty of crimes against humanity. Habré, president of Chad from 1982 to 1990, has been sentenced to life in prison in Dakar, Senegal, where he was tried. He is the first former head of state to be tried and found guilty of crimes against humanity in one country (Chad) by the courts of another country (Senegal).  Here’s more on the verdict from The Guardian. The verdict resonates especially with HRDAG because of our role in the trial. In September 2015, director of research Patrick Ball ...

Read More


The case of Ana Lucrecia Orellana Stormont

When working with documents in an archive, every document offers the opportunity for statistical study and quantitative research. But a document can also offer the discovery of a story. That is the case with the disappearance of Ana Lucrecia Orellana Stormont, who was reported missing on June 6, 1983, at the age of 35. Ana Lucrecia, a professor of psychology at the University of San Carlos, was scheduled to attend a meeting with Edgar Raúl Rivas Rodríguez at the Plaza Hotel in Guatemala’s capital city. Edgar, who also went missing, was a teacher at the School of ...

Read More


HRDAG Retreat 2016

What do you get when you bring seven statisticians, one quantitative political scientist, a writer, a computer scientist, and an administrator together for four days in a vacation rental on California’s Russian River? A lot of code, a technical paper and book chapter revised, another paper started, a great hike in the Redwoods, descriptions of food poisoning and crash landings in war zones, and a lot of talk about feelings. Did I mention feelings? On the first evening of the annual retreat of the Human Rights Data Analysis Group, executive director Megan Price asked ...

Read More


Data Archaeology for Human Rights in Central America: HRDAG Collaborates with UWCHR

Patrick Ball is kicking himself for a decision he made almost 25 years ago. “I was clever, but I wasn’t smart,” he says ruefully, as he considers the labyrinth of tables and ASCII-encoded keystrings he used to design a database of human rights violations for the pioneering Salvadoran non-governmental Human Rights Commission (CDHES). Now I’m sitting in his office in San Francisco’s Mission District watching over his shoulder, and trying to keep up, as he bangs out code to decipher the priceless data contained in these old files. Created in 1991 and 1992, ...

Read More


A geeky deep-dive: database deduplication to identify victims of human rights violations

In our work, we merge many databases to figure out how many people have been killed in violent conflict. Merging is a lot harder than you might think. Many of the database records refer to the same people--the records are duplicated. We want to identify and link all the records that refer to the same victims so that each victim is counted only once, and so that we can use the structure of overlapping records to do multiple systems estimation. Merging records that refer to the same person is called entity resolution, database deduplication, or record linkage. For ...

Read More


Welcoming a New Board Member

As we get ready to begin our fourth year as an independent nonprofit, we are, as always, indebted to our Advisory Board and to our funders for their support and vision. We’re finishing up a busy year that took us to Dakar (for the trial of former Chadian dictator Hissène Habré), Pristina (for the release of the Kosovo Memory Book), Colombia (for work on a book about the Guatemalan Police Archives), and kept us busy here at home working on police violence statistics. But one of our biggest victories has been to score a new, talented, wise Advisory Board member—Michael ...

Read More


An Award for Anita Gohdes

On November 26, HRDAG colleague Anita Gohdes was awarded the German Dissertation Prize for the Social Sciences. The patron of the prize is the President of the German Parliament, Norbert Lammert, who presented Anita with the award. Anita’s dissertation, “Repression 2.0: The Internet in the War Arsenal of Modern Dictators,” investigates the role played by social media networks in modern dictatorships, such as President Assad’s regime in Syria. On one hand, Anita argues, social media can help opposition groups to organize more effectively, but on the other ...

Read More


Changes at HRDAG

A special announcement from the HRDAG Advisory Board: Beginning officially on December 1, HRDAG is changing leadership. After nearly three years as Executive Director, Patrick Ball will become the new Director of Research. Megan Price will be the new Executive Director. Patrick has spent more than 25 years working at the intersection of human rights and statistical science. Over that time, he has finely honed statistical methodology for quantifying mass killings. Now he is excited about the chance to go deeper into research for HRDAG and develop additional approach...

Read More


Rapid response to: Civilian deaths from weapons used in the Syrian conflict

On November 4, 2015, the BMJ published our "Rapid Response" to Civilian deaths from weapons used in the Syrian conflict (BMJ 2015;351:h4736). The response was co-authored by Megan Price, Anita Gohdes, Jay Aronson (Carnegie Mellon University, Center for Human Rights Science), and Christopher McNaboe (Carter Center, Syria Conflict Mapping Project). We have three concerns about this article. First, the article apportions responsibility for casualties to particular perpetrator organizations based on a single snapshot of territorial control that ignores the numerous (and ...

Read More


HRDAG Testifies in Hissène Habré Trial

Last week HRDAG’s executive director, Patrick Ball, served as an expert witness for the prosecution in the trial of Hissène Habré, the ruler of Chad from 1982 to 1990. The trial is taking place in Dakar, Senegal, where the 73-year-old Habré has been living since 1990 when he fled Chad. He has already been sentenced to death in absentia in Chad. Habré is being charged with war crimes, crimes against humanity, and torture that took place during his eight-year reign. The trial is happening at the Extraordinary African Chambers, which was inaugurated by Senegal ...

Read More


Beka Steorts Named MIT Under-35 Innovator

We’ve known for years that Beka Steorts is on the cutting-edge of statistical science, and now The MIT Technology Review has realized the same. Last week she was named one of 35 Innovators Under 35, in the category of humanitarian. We first became familiar with Beka's work in 2013 when she was a visiting professor at Carnegie Mellon and was introduced to us by Prof. Steve Fienberg. Since then, we’ve felt very fortunate to collaborate with her on projects such as the UN enumeration of casualties in the Syrian conflict, and we look forward to many more years of work ...

Read More


Analysis of Homicide Patterns in Colombia

Last week Forensis, the Colombian National Institute of Forensic Medicine’s flagship publication, published the first of our analyses of homicide patterns in Colombia. Authored by HRDAG executive director Patrick Ball and UN colleague Michael Reed Hurtado, “Cuentas y mediciones de la criminalidad y de la violencia” (pages 529-545) explores, as the title suggests, the quality of “truth” contained within crime registries. Citing the problem of partial data, missing data, and inherent design bias, Patrick and Michael write that no register, official or unofficial, ...

Read More


Ten Years and Counting in Guatemala

We have accomplished so much in the last 10 years at the Historical Archive of the National Police. And yet, despite the efforts, dedication, and commitment of each person who since 2006 has worked in the AHPN, we still can not say “mission accomplished.” In 10 years the environment at the Archive has changed so much and become so full of life. Where the building once sheltered unknown stories, over time some of those stories have been revealed. But Guatemala has a long way to go in letting the world get to know more deeply about the secrets within the documents ...

Read More


Celebrating Ten Years of Data from the AHPN

Ten years ago, in July 2005, human rights officers stumbled upon a nondescript warehouse in a commercial zone of Guatemala City and changed history. They had discovered an archive–its existence kept secret–belonging to the Guatemalan National Police, whose officers committed human rights atrocities on behalf of the government during the civil war. Inside the building was the bureaucratic detritus typical of a large government agency: 80 million pages detailing shifts worked, tasks assigned, assignments fulfilled, workers’ whereabouts, and who was supervising ...

Read More


Update of Iraq and Syria Data in New Paper

This week The Statistical Journal of the IAOS published a new(ish) paper by Megan Price and Patrick Ball. The open-access paper, Selection bias and the statistical patterns of mortality in conflict, is a revisiting and updating of both the Iraq and Syria examples used in an earlier paper, Big Data, Selection Bias, and the Statistical Patterns of Mortality in Conflict, which was published last year inThe SAIS Review of International Affairs (JHU Press, 2014). HRDAG believes that the concerns highlighted by these examples are important for a wide variety of audiences, ...

Read More


Patrick Ball Honored with Degree at Claremont Graduate University

We’re happy to announce that our executive director, Patrick Ball, has been presented an honorary degree from Claremont Graduate University in Claremont, California. University President Deborah Freund presented the degree to Patrick at the university’s 88th annual commencement ceremony on Saturday, May 16, 2015. The degree conferred was Doctor of Science honoris causa. “We at CGU are thrilled that Patrick Ball accepted our Honorary Degree invitation and joined us for commencement,” said Thomas Horan, CGU Professor and Director, Center for Information ...

Read More


Our work has been used by truth commissions, international criminal tribunals, and non-governmental human rights organizations. We have worked with partners on projects on five continents.

Donate