paper web signal

Audit of 40 open-weight LLM reports finds 1 full disclosure

TL;DR

  • Of 40 sampled technical reports for new 2025 open-weight LLMs, only 1 disclosed data mixture, token count and compute together.
  • Of 48 new architectures added to Hugging Face transformers in 2025, 39 take at least two of three core blocks from another model, most often Llama.
  • Only 19 of 73 architectural or training novelty claims came with a controlled ablation, and 4 of 40 reports stated more than one training run.

One of 40 technical reports for new open-weight large language models released in 2025 disclosed its data mixture, token count and compute together, according to an arxiv audit by Bahaeddin Türkoğlu.

The audit started from the 48 new LLM architectures added to the Hugging Face transformers library in 2025 and sampled 40 of their accompanying technical reports. Structurally, the "new" label is thin: Türkoğlu writes that "39 of the 48 architectures added in 2025 take at least two of their three core blocks from another model, most often Llama, although part of this reflects how the library writes code."

Testing of the claimed innovations is thinner. The paper reports that "19 of 73 architectural or training novelty claims come with a controlled with/without comparison," that "4 of 40 reports state more than one training run," and that "8 rerun the models they compare against under one setup." The remaining reports lean on numbers reported elsewhere rather than head-to-head reruns.

The paper is careful on its own claim, noting that "an untested claim may still be true," and proposes a set of reporting standards for future open-weight releases.