New Paper Explains Why SFT Breaks Multi-Task LLMs, Not RL

A new arXiv paper called "SFT Conflicts, RL Coexists" argues that the default recipe for stacking capabilities into an LLM, round after round of supervised fine-tuning, is quietly sabotaging itself, and that reinforcement learning avoids the same trap for a specific mathematical reason.

The setup is a controlled multi-task run across math (MATH500), science (MMLU), logic (Knights & Knaves) and code (LiveCodeBench). When the authors train the tasks sequentially with SFT, average performance collapses to 23.1% below the base model. Mixed-data SFT recovers, netting a 7.4% gain. Multi-stage RL, by contrast, comes out 24.9% ahead. That is a large gap for a paper whose real contribution is the explanation, not the raw numbers.

The mechanism claim is where it gets interesting. The authors trace the difference to the shape of the parameter updates each method produces. SFT rewrites parameters aggressively. The paper reports an L₂ update norm around 7.4, with 93% of parameters moving by more than 10⁻⁵, and cross-task cosine similarity near 10⁻¹. RL produces updates with an L₂ norm around 3×10⁻², where only 20% of parameters cross that threshold, and cross-task cosine similarity falls to roughly 10⁻⁵. In their framing, "interference in SFT is norm-limited, scaling with the absolute gradient magnitude, whereas interference in RL is variance-limited, bounded by the gradient variance induced by advantage normalization and on-policy optimization." Sparse, near-orthogonal updates mean tasks do not step on each other.

Their proposed remediation is called Parallel-RL: run RL on each task in its own process, then merge the resulting deltas. Variants range from a naive sum or mean, to TIES/SVD-style sparse merging, to a small post-merge adaptation pass on 5% of the original training data. The pitch is that RL's orthogonality is exactly what makes cheap model merging viable.

There are things the paper does not settle. The four-benchmark suite is narrow, and it is a single research group's controlled experiment rather than a cross-lab reproduction. The checkpoints trained here are not the same beast as production post-training pipelines that already blend RLHF, DPO, rejection sampling and curriculum. Whether the norm-versus-variance decomposition survives once you scale rollout groups, swap reward models, or add off-policy corrections is genuinely open.

If the story holds up on bigger stacks, though, it reframes the usual "SFT for behaviors, RL for preferences" division as something closer to "SFT bakes in interference, RL does not," and hands teams a specific reason to try merging RL-trained specialists instead of committing to one monolithic multi-task fine-tune.