Hugging Face crowd reproduces 2,226 ICML papers; 23% contested

Hugging Face ran a 19-day hackathon between July 15 and August 2, 2026, in which more than 1,200 community members used coding agents to reproduce papers from ICML 2026. According to the Hugging Face post, 2,226 papers were attempted, or 34% of the conference's 6,352 accepted papers, and 35,908 individual claims were judged. The headline verdict: "23% of examined papers (496) had at least one claim falsified or contested," with "242 papers where independent teams reached opposite verdicts on the same claims."

The falsifications get specific. In "Towards Optimal Robustness in Learning-Augmented Paging," the paper's H_k + O(1) robustness bound came apart once teams pushed measurements out to k = 1,024; the true additive term grows roughly like 0.38 ln k, with "confirmed growth at roughly nine sigma." Three teams independently produced counterexamples to a convex-hull claim in "Attention's forward pass and Frank-Wolfe," with violations logged at t = 224, ~3,800, and 6,416 steps. The authors confirmed the same day. In "Do Transformers Need Three Projections?", reproducers found that "~66% of evaluated label positions were EOS padding tokens training to near-zero loss," turning the abstract's advertised 3.1% quality cost for a 50% cache reduction into roughly 9.4% once corrected.

One line from the challenge notes lands harder than the numbers. A reviewer on an accepted ICML 2026 spotlight paper wrote, "My low confidence score is because I did not check all the proofs carefully." The paper was later found during the hackathon to contain a falsified mathematical claim.

The exercise is not clean, and Hugging Face says so. The judge is itself an LLM, an automated Logbook Judge running the open-weights GLM-5.2 model, and one participant's logbook wrongly declared a paper's method 2x slower than baseline because of a per-trajectory versus per-batch arithmetic bug in the reproduction. Corrected, it confirmed the paper's claimed 8x speedup. Two arXiv corrections are in flight.