P-hacking is a long-standing problem in science, and LLMs make it worse: as tools for annotation or LLM-as-a-judge, they allow tuning parameters until a desired result appears. We propose a way to mitigate this problem, and conduct a preregistered study of its effectiveness! a…
AI Weekly's analysis
→
- A new arXiv paper proposes preregistering LLM experiments and running the confirmatory analysis on the first eligible model released after registration.
- Across 20 models from four providers and 11 configurations, the protocol blocked p-hack transfer in 73.9% and 72.7% of cases across two tasks.
- The authors preregistered their own experiment; of 7 configurations that hacked the prior model, 6 failed to carry over to the next.
Read full analysis →