Tsinghua's SEED Beats GRPO by up to 45.9 Points on ALFWorld

Outcome-only reinforcement learning has been the workhorse behind the current wave of agent training, but it has an obvious hole. A trajectory-level reward tells you the episode succeeded or failed, and nothing about which of the fifty tokens along the way actually mattered. A new Tsinghua-led paper on Hugging Face, titled SEED (Self-Evolving On-Policy Distillation), takes a swing at that gap by getting the policy to critique itself.

The setup is two-stage. First, a supervised fine-tune where an external analyzer, GLM-5.2, labels 1,440 offline trajectories with what the authors call hindsight skills, and the policy learns to produce those skills from completed rollouts. Then, during RL, the same policy checkpoint plays two roles at once: it acts, and it re-analyzes its own trajectories to extract skills, which are folded back in as a dense token-level distillation signal on top of GRPO. Nothing extra runs at inference; the skills only show up during training.

The numbers the authors report are the reason this is getting attention. Across Qwen2.5-3B, Qwen2.5-7B and Qwen3-1.7B backbones, SEED beats GRPO on ALFWorld by 14.9 to 45.9 macro-average points, on WebShop task-completion score by 8.7 to 19.8, and on search-based QA by 1.4 to 9.3 points. The Qwen3-1.7B result is the eye-catcher, jumping from 46.1 to 92.0 on ALFWorld. Sample efficiency also holds up in the paper's telling: SEED at 60% of the ALFWorld training set scores 80.7, above GRPO's 75.0 on the full set.

The honest caveat is that Stage 1 leans on a capable external analyzer to bootstrap the skill vocabulary, so labs without something like GLM-5.2 on hand are not quite getting a free lunch. The reporting also does not tell you how the self-analysis loop behaves past 150 update steps, at frontier scale, or when the extracted skills are simply wrong. The paper's own ablations show that a naive skill-prompt variant, Skill-GRPO, actually regresses ALFWorld from 75.0 to 60.2 on Qwen2.5-3B, so the gains are not just from adding skill text to the context.

If the result holds, the interesting downstream is for teams training small agent models on constrained compute, since the whole recipe adds dense supervision without a bigger reward model, an external critic, or any inference-time overhead.