PCSD paper claims RL boost for language agents on ALFWorld

A new paper on arxiv proposes a technique called PCSD, short for Persistent Consistency for Self-Distillation, aimed at one of the more stubborn problems in training language model agents with reinforcement learning: sparse rewards that only arrive at the end of a long chain of actions.

The method, described on arxiv, tries to combine two signals that are usually kept separate. On one side, what the authors call dense token-level supervision from a privileged teacher model. On the other, the sparse environmental reward that says whether the agent's whole trajectory succeeded or failed. Their argument is that prior approaches which look at token-level differences in isolation are noise-sensitive, and approaches that assign a single weight to a whole step overlook positional variation in how reliable the teacher actually is.

PCSD's three moving parts, per the abstract, are adaptive windows with exponentially decayed aggregation, trend-aware modulation to reduce declining-support effects, and sigmoid gating for continuous weight generation. The whole thing is optimized jointly with GRPO. On ALFWorld, the paper claims 15.6-13.3 point gains over GRPO alone and 6.2-5.5 points over a baseline called SDAR, with a 15.8 point gain on the unseen split. On WebShop the method reportedly stays competitive with existing baselines rather than pulling ahead.

The honest caveat is that the abstract does not give you the compute budget, the base model, ablations that isolate which of the three components actually does the work, or whether the privileged teacher needs to be substantially bigger than the student. Those matter because the interesting question with self-distillation is always whether the gain survives when the teacher is not a much larger, much more expensive model. The WebShop result, where PCSD only matches rather than beats baselines, is another reason to read the specifics as reported, not settled.

If the results hold up outside of the two environments tested here, the direction is worth watching, because credit assignment across long agent trajectories is exactly the bottleneck that current agentic RL keeps running into.