Tech Note on CLEAN: Teaching a small classifier to spend expensive attention wisely
The California Law Enforcement Accountability Network (CLEAN) initiative works with millions of pages of records released under California’s police transparency laws. Unfortunately, those pages do not arrive as tidy, one-document-per-file PDFs. A single file is often a stack of concatenated documents: reports, complaints, interview transcripts, and administrative memos scanned together, with the notice of complaint buried somewhere inside. The notices also are not uniform: their layout varies from agency to agency, with no single template to match against. The task is not “which files contain a notice” but the finer-grained “which page is the first page of a subject notice.”
We could try to read everything, but the corpus is far too large for exhaustive human review. We could ask a large language model about every page, but at millions of pages that is prohibitively expensive and slow. So from the start the real question was not whether to use LLMs, but where to spend them, and where to spend the even scarcer resource: human attention.
The way to spend that attention well is to not spend it on every page. A trained classifier can read every page instead. The problem is simple enough because each page is a binary yes-or-no question, and the notices share enough common language for a bespoke classifier to learn from. So we built one in a teacher-student setup: two LLM teachers produce labels, and a small student model learns from them to become a cheap classifier you could deploy.
Read the full tech note here: The California Law Enforcement Accountability Network (CLEAN) initiative: Teaching a small classifier to spend expensive attention wisely