ContinualSkillBench: LLM skill libraries mostly match prompts

There is a paper out this week that pokes a hole in one of the more architecturally satisfying ideas in agent design: give a language model an external skill library, let it write reusable procedures as it goes, and it should get better at future tasks by pulling from that store. The new benchmark, ContinualSkillBench on arXiv, tests whether that actually happens, with Muhan Zhang of Peking University among the corresponding authors.

The setup is deliberate. Five representative domains, one hundred interconnected subtasks per domain, ordered by increasing difficulty and designed with opportunities for cross-task skill reuse. The authors then compare agents that maintain explicit, saved skills against ones that just use plain in-context learning over prior task feedback. Sequential execution generally improves performance in both cases, but the punchline is that in-context learning performs comparably to explicit skill maintenance on average, meaning much of the improvement arises from adaptation to prior context and feedback rather than reusable skill abstraction alone.

Explicit skills are not useless. The paper finds they still provide selective benefits for tasks requiring reusable procedures or precise outputs, so the skill-library idea holds up in niches where an exact procedure matters. But the same paper reports that less capable models tend to accumulate larger, more fragmented collections of task-specific skills, which is roughly the opposite of what a skill library is supposed to do. Weaker models are worse at abstracting, so their libraries bloat.

For anyone building agent products, the practical read is that architectural complexity around skill storage may not be paying for itself compared with feeding good recent context into the prompt. That reframes a lot of the self-improving agent pitch: what teams are seeing may be short-horizon in-context adaptation dressed up as skill acquisition.

The honest caveats are worth stating. What the retrieved abstract does not give you is the list of which specific LLMs were evaluated, which five domains were used, or what exact skill-library framework was the comparison target, so take the comparability finding as an average across the reported setups rather than a universal verdict. And an academic benchmark, however carefully constructed, is not a production workload where a curated skill store might still earn its cost.

If the finding holds up as others reproduce it, the upside is on the simpler side of the stack: teams that resisted the skill-library detour, and platform owners whose infra bill improves when agents stop paying retrieval-plus-store overhead on every step.