Purpose
On a regular basis, judges lacking technical expertise admit the results of under-tested, poorly trained algorithms as evidence in trials across the United States, and defense attorneys usually don’t have the resources to challenge them. The longer these tools avoid regulation or cross-examination, the more likely it is that AI-based systems will generate inscrutable conclusions with a high-tech veneer.
While much of the training data and code is hidden behind NDA and trade secrets, what we have seen so far is deeply troubling. A leading DNA-matching system was trained on clean lab samples, but is employed to draw conclusions from messy crime scenes. A gunshot detector’s training data is based on what people thought sounded like a gunshot, rather than on ground-truth labels. Data science teams in the private sector wouldn’t accept such a disconnection between training and application, but our courts don’t know any better.
“Scientifically valid and accurate forensic science strengthens all aspects of our justice system.”
Memorandum of Understanding between the Department of Justice and the National Institute of Standards and Technology, 2015.
We will use the adversarial court system to learn more, and to educate. When we join cases as an expert witness, defense attorneys will use the discovery process to compel third-party providers to let us evaluate their data and code. We have strong reason to believe we’ll uncover a lot more that we can educate courts about, both in specific cases and more broadly through continuing education, conferences, and research papers. Plus, practically, prosecutors will eventually rely less on systems whose damaged credibility imperils their conviction rates.
The picture we paint here is of criminal trials, but the premise extends everywhere the government relies on software it doesn’t understand to come to conclusions that affect people’s freedom and finances. We’re starting here because meaningful legislative action seems quite unlikely and executive-branch functions at the federal and state level are increasingly fueled more by motivated reasoning than a sense of impartial accountability to the people — but the adversarial system of the courts, frustrating and imbalanced as it may often be, offers the mechanisms to make real change purely through being on the right side of justice.
