Researchers ship AI brief retrieval for NJ public defenders

A paper out of a partnership with the New Jersey Office of the Public Defender lands on a claim that is easy to skim past and worth pausing on. The team behind NJ BriefBank says the retrieval benchmarks the field has been using for legal AI do not, in fact, transfer to the actual research a public defender does.

That is a more pointed finding than the usual results-improved abstract. If you are building or buying legal AI for public interest work, the leaderboard your vendor is quoting may have very little to do with what a defender types into the tool the night before a hearing. The authors, Dominik Stammbach, Kylie Zhang, Patty Liu, Nimra Nadeem, Inyoung Cheong, Lucia Zheng and Peter Henderson, argue the fix is domain knowledge rather than bigger models: query expansion with legal reasoning, domain-specific data, and curated synthetic examples all lift retrieval quality on their evaluation set.

The stakes here are less about a new model and more about who gets to measure it. Public defense sits at the intersection of a constitutional right to counsel, overwhelming caseloads, and constrained resources, and the authors' pitch is that AI tools are being suggested as solutions for those workloads without much evidence they meaningfully help defenders' day-to-day work. To push back on that, they are releasing a taxonomy of realistic defender search queries and a manually annotated evaluation dataset, and they report the public benchmark is highly correlated with a proprietary retrieval dataset that experienced public defenders annotated. That is the useful bit for anyone else working in this space.

The honest caveat is what the abstract leaves out. It does not give retrieval numbers, model comparisons, dataset sizes, or a picture of how the tool is actually being used inside the NJ office. What it does give is a public target that legal-tech vendors and academic labs can now aim at, which is a healthier baseline for the field than the current pattern of quoting benchmark wins that do not survive contact with a real caseload.