CMU paper: preregister LLM studies against the next unreleased model

There's a quiet methodology problem sitting under a lot of LLM research, and a new paper from CMU tries to defuse it with a trick borrowed from clinical trials. Because researchers can retune prompts, decoding parameters, and output formats until a hypothesis pops out, LLM-based studies are unusually easy to p-hack. Maria Thomas, Kristina Gligoric, and Nihar B. Shah propose the obvious fix that nobody had operationalized: commit to the analysis, then run it on a model that doesn't exist yet.

The protocol, laid out in an arXiv preprint, asks researchers to finalize their procedure on today's models, preregister the plan together with a set of eligible future models, and then run the confirmatory analysis on the first eligible model released afterward. Because that target doesn't exist at commitment time, it can't be hacked against. The empirical claim underneath is that configurations that hack one model often fail to transfer to the next.

The numbers back the intuition, though not overwhelmingly. Across 20 models from four providers and 11 LLM-analysis configurations, the protocol would have blocked successful transfer of the p-hack in 73.9% and 72.7% of cases across two tasks. The authors also ate their own cooking: they preregistered their experiment, and of seven configurations that hacked the prior model, six failed to carry the hack to the eligible next release.

The honest caveat is that roughly a quarter of hacks still transferred, so this is a filter rather than a fix, and the paper doesn't tell you what makes a hack sticky enough to survive the model switch. What the reporting also doesn't give you is how the protocol behaves in fields where no eligible model ships in a reasonable window, or where 'the same result' is hard to define, like open-ended qualitative work.

The useful reading for anyone running LLM-based measurement pipelines is that a lightweight, verifiable commitment device now exists. Reviewers can ask for it, journals can require it, and small labs that can't afford sprawling multi-model sweeps get a way to make a validity claim without one. Take the specifics as reported, not settled, but the direction is the part worth watching.