Function-Aware FIM Adds Up to 5.4 SWE-Bench Points to Qwen

Ordinary code pretraining teaches a model to read source left to right, but a coding agent at deployment runs a different loop: it emits an action, waits for a tool return, and continues reasoning around that value. A new arXiv paper from Yubo Wang and colleagues argues that this loop is structurally the same object as a function call site, where a caller binds arguments, a callee returns a value computed elsewhere, and downstream code consumes it. That conditioning structure, they say, already exists at internet scale in ordinary code, and the fix is to exploit it as a mid-training step between base pretraining and standard agentic post-training.

The recipe is called function-aware fill-in-the-middle. Function bodies are picked for masking using program dependency graph analysis and what the authors describe as a complexity-inferability double criterion, and the model learns to reconstruct them from surrounding code. The mid-training corpus is 2.6 billion tokens drawn from 968 GitHub repositories, decontaminated against the target benchmarks, and applied to Qwen2.5-Coder-Instruct at 7B and 14B and to Qwen3-8B before existing post-training pipelines. Reported gains on SWE-Bench-Verified are +2.8, +3.0 and +3.2 for those three models; on SWE-Bench-Lite the same models pick up +3.7, +4.0 and +5.4. The paper says the improvement holds across two post-training pipelines, R2E-Gym and SWE-Smith, and on a non-Qwen2.5 base via SWE-Lego.

The reason this is worth reading if you are not training foundation models yourself is that mid-training sits in the cheap slot where labs actually differentiate their coder products, and a described-in-detail bolt-on that adds several SWE-Bench points on top of existing post-training will be copied quickly. The authors also report that the usual capability erosion agentic post-training inflicts on non-agent coding (LiveCodeBench) and non-coding tool-use (tau-bench, BFCL) is mitigated, and that a Python-only corpus generalizes to non-Python tasks because the function-call inductive bias survives post-training.

The honest caveat is that every number here is self-reported on Qwen family bases, and independent replication is what would tell you whether the lift survives once other labs and other post-training stacks are in the mix. What the abstract does not give you is the compute cost of the mid-training pass, or how sensitive the effect is to the exact function-selection criterion. For anyone shipping open-source coding agents, this is a specific, low-friction step to try before the next post-training run, and the first labs to integrate it cleanly are the ones that will show up on the next leaderboard.