New arXiv paper: 'meek models' close gap only on bounded metrics
TL;DR
- A new arXiv paper argues whether small AI models eventually catch up to frontier systems depends entirely on which performance metric you pick.
- The authors show validation loss gaps shrink, but on other unbounded metrics frontier models grow their lead forever with more compute.
- Bounded and unbounded metrics can suggest opposing policy responses for capabilities like software engineering, synthetic biology, or rhetorical persuasiveness.
Whether small, budget-constrained AI models will eventually catch up to frontier systems, or fall further behind forever, isn't a single empirical question. According to a new paper on arXiv by Alex Fogelson, Zachary A. Brown, Hans Gundlach, Jayson Lynch and Neil Thompson, accepted at the 2026 ICML Technical AI Governance Research Workshop, it depends entirely on which metric you are looking at.
The authors' framing is that performance metrics fall into two mathematical families with respect to compute. On validation loss, they report the gap between frontier and small models is shrinking. On other metrics, in the paper's own words, "frontier models grow their lead forever." They provide a tight mathematical condition: bounded performance metrics always eventually favor smaller "meek" models, while unbounded ones perpetually reward whoever spends more on compute.
Why this matters for anyone reading policy proposals is that bounded and unbounded metrics can point at opposite responses to the same capability. Measure a risky capability like software engineering, synthetic biology, or rhetorical persuasiveness on a bounded metric, and the story looks like democratization, with frontier-level ability proliferating through cheap open models into many hands. Measure the same capability on an unbounded metric, and the story flips to concentration, with the sharpest ability locked behind whoever can afford the largest training run. The authors argue that determining the apt metric for a domain is a prerequisite for policy, not a downstream detail.
The honest caveat is that this is a theoretical paper. It gives the mathematical conditions and the classification, but it does not tell you which specific real-world capability sits on which side of the line, and the authors themselves flag that many common bounded metrics have closely-related unbounded counterparts and vice versa. What the reporting does not give you is an empirical audit of the benchmarks currently in use, or which side of the line software engineering ability actually lives on today.
The forward-looking piece is that the useful next work is that audit. For open-model teams and smaller labs, framing the debate around bounded metrics is now a defensible strategic move rather than wishful thinking. For governance, the practical implication is that the metric argument has to happen before the threshold argument.
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: Two AI Metrics Diverged: Will it Make All the Difference?