Aspire benchmark: frontier LLMs stumble at open-ended self-improvement

The strongest evolved system in a new benchmark for open-ended self-improvement still trails a hand-engineered baseline. In a paper posted this week on arXiv, a team led by Yuhao Wu introduces Aspire, a benchmark in which an LLM agent is given only a natural-language capability goal — "become a better physicist" is the paper's own example — and must decide for itself what to learn, what data to train on, when to evaluate, and whether it has actually improved. The evaluation set stays hidden: 520 expert-authored items across six goals, from scientific and academic reasoning to mathematical reasoning to academic and scientific writing.

Under vague-goal prompting, the frontier models the authors tested scored well below their explicit-task references. Claude Opus 4.8 reached 27.07 against a 32.90 reference from PostTrainBench; GPT-5.6 reached 29.58 against 36.23. Agent-harness self-evolution did not close the gap either: "all three valid successors remain numerically below the original Qwen-Agent, which obtains 28.64 task macro and 27.65 example micro," the paper reports.

The problem: continued search often made things worse. "Agents often train on mismatched data and trust narrow self-evaluations, so local gains fail to transfer to hidden evaluation and continued search and training can erase earlier improvements," the authors write. Weight-level improvements, in their words, "remain sparse and unstable."

The paper does not break down which of the six goal areas the models struggled with most, and the harness-evolution numbers come from a small set of three successor systems. On this setup, current frontier models can execute a specified task competently and stall when they have to specify one for themselves.