Nemotron team opens the midtraining recipe for coding agents

An open recipe posted to arXiv spells out the 'midtraining' stage that frontier labs now insert between pretraining and post-training but rarely document. On Nemotron 3.5 Lightning, 500B midtraining tokens lifted SWE-bench Verified from 60.7 to 64.5, SWE-bench Multilingual from 55.8 to 62.8, and SWE-bench Pro from 32.0 to 35.5 after an identical SFT run, according to the abstract.

The paper, led by Soumye Singhal with thirteen co-authors, trains at 256K context on a mix of 'packed repositories, pull requests, commit histories, and build logs together with a small share of agentic traces.' On the larger Nemotron 3.5 Super, 280B midtraining tokens pushed DeepSWE, which the authors describe as 'a harder benchmark of long-horizon tasks,' from 11.7 to 15.7, while SWE-bench Verified moved only 72.9 to 74.3.

The authors frame the contribution as a disclosure problem: 'Frontier models, general-purpose and domain-specific alike, now include a midtraining stage between pretraining and post-training, but little has been published about how this stage is built.' Their stated rationale for inserting the stage is that pretraining 'sees almost no agent interactions, while SFT data is mostly synthetic, and very long SFT stages can hurt the model's later RL trainability.'

The abstract reports gains across both model sizes and claims midtraining 'improves robustness across agent harnesses on both models,' but publishes no per-harness numbers, and the composition of the agentic trace portion is not described.