Off-Policy 'Environment Evolution' Adds 18 Points on Terminal-Bench

A paper posted to arxiv this week proposes training terminal agents against environments that grow harder "generation by generation" instead of being synthesized on-policy from the model's own failures. The authors call the method environment evolution, and they report that long-horizon RL using it lifts Qwen3.6-27B by 14.4 percentage points and Qwen3.6-35B-A3B by 18.0 percentage points on Terminal-Bench 2.1.

The pitch targets a specific failure mode in recent co-evolution work: as the base model gets stronger, on-policy environment synthesis stops producing challenging enough scenarios. "Their dependence on on-policy rollouts limits generalization and the continuous provision of learning signals as the model becomes stronger," the authors write. Their alternative derives "three evolution directions that influence environment difficulty from the multi-turn learning objective" and implements them through what the paper describes as a "loop-engineered multi-agent harness."

Difficulty of the generated environments is validated in rollout experiments with Hy4 preview, Claude Opus 5 and GPT-5.6 Sol; the abstract keeps the frontier-model comparison qualitative and reserves the numeric gains for the two Qwen variants. The work sits inside a longer run of open-model RL papers moving through the alerts we track in agents and open source.