SKT turns 2,000 Agent Skills into 27,164 verified SFT runs

A quieter result inside the new SKT paper matters more than the headline number. The researchers ran their pipeline twice, once with the full verifier stack and once with the verifier and repair steps stripped out, and the raw-synthesis run made models worse on all four evaluation benchmarks. Verified trajectories, by contrast, lifted every one of them, with the gaps between the two data sources sitting at 11.91 to 24.61 points.

The pipeline itself, described in the paper on Hugging Face, starts from 2,000 distinct skills in the public skills.sh library, synthesizes 4,000 accepted task packages (1,520 single-skill, 1,295 two-skill, 1,185 three-skill) and then farms out 32,000 candidate task-teacher-harness combinations to four teacher models (MiniMax-M2.5, GLM-5, Qwen3.5-397B-A17B, and DeepSeek V4 Pro) running inside DeepAgents and OpenCode. Verification keeps 14,277 DeepAgents rollouts and 12,887 OpenCode rollouts, for the 27,164 total the abstract advertises.

Supervised fine-tuning of Qwen3.5-9B and Gemma 4 E4B-IT on that corpus improved every one of the 16 backbone-harness-benchmark comparisons the authors ran, with absolute gains from 3.20 points at the low end (Gemma 4 E4B-IT with OpenCode on SkillsBench, 7.08 to 10.28) to 18.91 at the high end (Qwen3.5-9B with DeepAgents on SkillEval, 51.62 to 70.53). Withholding the skills at inference time cut the gains back to 0.53 to 5.69 points, which the authors read as evidence that SKT is teaching models to use externally supplied skills rather than internalizing them. Scaling the training pool from 100 to 2,000 skills raised SkillEval monotonically from a 55.24 baseline to 72.48.

The honest caveats are worth stating. Cross-harness transfer retained only about half of the matched-harness benefit, so a checkpoint tuned on one runtime is not a drop-in replacement for another. SkillEval was constructed using the same synthesis pipeline as the training data (from a disjoint skill pool), which is the right way to build a scalable held-out benchmark but leaves open whether third-party evaluators will see the same lift. The paper also does not report the compute cost of running the pipeline, and it does not benchmark against the reinforcement-learning-from-execution approaches surveyed in the related work.

The useful takeaway for anyone building on top of a growing Agent Skills library is that verification is not a nice-to-have inside a synthetic-data loop. The alternative is not neutral; it is a regression.