LLM Report Audit: 22 of 40 Copy Benchmark Scores, 1 Discloses Training Data in Full

Found first: a primary source the press has not covered yet.

Bahaeddin Türkoğlu audited 40 LLM technical reports and found that 22 took some or all benchmark comparison scores from other papers rather than re-running them, and only 1 disclosed data mixture, token count, and compute together. The paper also examined architecture code for 48 models added to Hugging Face transformers in 2025 and found 39 share at least two of their three core building blocks with an earlier model, most often Llama.

What the source says

Türkoğlu drew a random sample of 40 technical reports and analyzed novelty claims and reproducibility disclosures, with two independent coders. Of 73 architectural or training novelty claims, 19 included a controlled ablation, and only 4 reports described more than one training run. Eight ran comparison models under a single evaluation setup, and 8 included any third-party evaluation. The author notes that an untested claim may still be true, and that copied scores are sometimes unavoidable when a comparison model cannot be re-run. The paper proposes seven minimum reporting rules and releases code and data publicly.

Why it matters

Capability rankings from LLM technical reports shape which models practitioners adopt and fund. When 22 of 40 reports use benchmark scores drawn from different evaluation setups, comparisons across papers are not grounded in a shared baseline. The architectural analysis adds a separate issue: 39 of 48 models in the Hugging Face library share the same core structure as earlier ones, yet the reports rarely test which parts are new. For anyone evaluating LLM claims without access to the original training runs, the paper makes explicit how much is unverifiable.