LLM Report Audit: 39 of 48 Architectures Recycle Core Blocks, 1 of 40 Discloses Training Data

Found first: a primary source the press has not covered yet.

A new audit of LLM technical reports finds that 39 of 48 architectures added to Hugging Face's transformers library in 2025 reused at least two of three core building blocks from earlier models, predominantly Llama, and that only 1 of 40 sampled reports simultaneously disclosed data mixture, token count, and compute. The paper, by Bahaeddin Türkoğlu, is available on arXiv.

What the source says

Türkoğlu drew 40 technical reports from the 48 new LLM architectures added to Hugging Face transformers in 2025 and coded each against three dimensions: architectural novelty, ablation quality, and training transparency. Two independent coders worked from script-verified quotations. Of 73 novelty claims examined, 19 included a controlled ablation. Only 4 of 40 reports described more than one training run. Only 8 of 40 reran baseline models under identical conditions. The paper proposes seven minimum reporting rules, releases code and data under CC BY 4.0, and states: "We do not judge model quality, and an untested claim may still be true."

Why it matters

Benchmark comparisons from LLM technical reports inform research and procurement decisions across the industry. When only 8 of 40 reports reran any baselines under consistent conditions, those comparisons cannot be verified from the reports themselves. The 73 novelty claims make the scope of the problem concrete: 54 were made without a controlled test. The seven proposed rules define what a verifiable report would need to include.