Paper Maps LLM Long-Horizon Planning Across Training Stages

There's a persistent question in agentic AI: when a model fails at a multi-step task, is it a scaling problem, a reinforcement learning problem, or something the model was never taught in the first place. A paper posted to arXiv on July 27 by researchers at the Chinese Academy of Sciences' Institute of Automation tries to answer that by running a controlled study across all three training stages, and the picture it paints is that data choices in pre-training determine far more of the ceiling than the RL step that gets most of the attention.

The headline pre-training finding is that models trained with an explicit chain-of-thought that simulates state transitions, an internalized world model, generalize much better to long-horizon planning than models trained to predict actions directly. On long tasks the pass@8 gap is reportedly 42.9 points. Data mix matters too. With only atomic skills in the corpus, long-horizon pass@8 sits at zero, but mixing in just 5% of long-horizon trajectories lifts middle-horizon pass@8 from 0.83% to 51.88%. And suboptimal trajectory data poisons the well. A 4-optimal to 8-suboptimal ratio collapses long-horizon performance to near zero because errors accumulate and amplify across turns.

The post-training analysis is where the paper gets more prescriptive. It frames the problem in mutual-information terms and argues that GRPO, the sparse-reward RL method behind a lot of recent agent training, breaks down in what the authors call the 'unsupported region': long-horizon tasks where credit assignment is too coarse. On-policy distillation (OPD), which passes dense, token-level signals from a teacher, holds up in that region. The stated gradient-direction similarity to the final checkpoint is 0.0-0.3 for GRPO versus 0.6-0.8 for OPD.

For multi-teacher setups, the paper's MOPD framework sorts outcomes into three modes: shared and compatible patterns give cross-environment generalization; shared but conflicting patterns give continual learning with some forgetting; non-shared conflicting patterns cause catastrophic forgetting, with Fantasy Alchemy accuracy in one experiment falling from 94.14 to 15.62 after switching teachers.

The honest caveat is that all of this runs on a synthetic three-domain benchmark (Fantasy Alchemy, Livestock Farming, Electronic Assembly), a 1.2B-token corpus, and Qwen2.5 as the base, not a frontier-scale training run, and the paper does not claim its numbers will transfer verbatim to production agents. What the reporting doesn't give you is any comparison to closed-lab pipelines or evidence about how the code and datasets on the project's GitHub will be maintained. Still, for teams building open agent models on smaller budgets, the recipe on offer, world-model CoT plus a small but non-zero long-horizon share plus OPD over GRPO in the hard regime, is unusually specific for a field that mostly ships vibes.