Kierston Paper Tests Sixteen LLMs as Allentown Investigators
TL;DR
- Kierston simulates roughly 125,000 Allentown, Pennsylvania residents and their telecom, financial, administrative and license-plate records to evaluate LLM investigators.
- KierstonBench spans 32 task families covering record linkage, routine reconstruction, incident investigation and assessment of deceptive accounts.
- Of sixteen models tested, most submitted identifications the records did not support; a small frontier group abstained but still none replaced an investigator.
Two researchers built a synthetic Allentown, Pennsylvania of about 125,000 residents, piped out their telecommunications, financial, administrative and license-plate records, and asked sixteen language models to investigate.
The paper introduces a simulation called Kierston and a benchmark called KierstonBench, structured around 32 task families "covering record linkage, routine reconstruction, incident investigation and assessment of deceptive accounts."
The headline finding is blunt. "In an evaluation of sixteen language models, most could not perform records investigation and submitted identifications the records did not support," authors Andrew Semenov and Svyatoslav Dorofeev write. A smaller set of frontier models solved most of the tasks "and abstained when evidence was insufficient." Even those, the paper reports, "still fell for common deceptions and assumptions, and none replaced an investigator."
The abstract names no model and publishes no per-system numbers. The code sits on GitHub. It arrives into the 47 surveillance-adjacent alerts we've tracked in the last 90 days.
Originally reported by huggingface.co
Read the original article →Original headline: Arxiv 'Kierston' Simulates Allentown PA and Benchmarks 16 LLMs on Surveillance Tasks