CAST uses puzzle-game solvers to teach LLM agents turn by turn

Reinforcement learning for language agents keeps hitting the same wall: the reward only arrives at the end of a long trajectory, so the model has almost no idea which of its fifty turns actually mattered. A new paper from Yu Wang and collaborators, posted to arXiv as "CAST: Game Solvers as Turn-Level Teachers for LLM Agents," proposes an unusually direct fix. Instead of hand-crafting per-step rewards or training a separate value model, they let an off-the-shelf game solver grade every move.

The mechanic is simple. For a game where a solver exists, you can ask that solver how good the current state is, take the action the LLM proposes, and ask again. The delta becomes a turn-level advantage signal that gets injected into standard RLVR training. The authors argue that under a soft-optimal solver assumption, maximizing that advantage is equivalent to on-policy distillation from the solver, but only requiring scalar values rather than teacher logits, which is a much cheaper thing to consume.

The headline results, taken from the paper's own abstract, are two-fold. On the three puzzle domains they train in, Sokoban, Minesweeper, and Rush Hour, CAST reportedly outperforms all trained baselines under both in-domain and unseen-difficulty evaluation. More interesting is the transfer claim: the same trained policy achieves the highest average zero-shot performance on ALFWorld and WebShop, which are not games at all but household-simulation and web-shopping agent benchmarks. If that generalization is real, it suggests solved puzzles can act as a general-purpose gym for agent skills, not just as a puzzle-specific curriculum.

The honest caveats are the ones you would expect. This is one paper, one set of authors, one benchmark suite, and the whole approach is bounded by domains where a decent solver already exists. What the write-up does not give you is a clean picture of margins over specific baselines, the base model sizes involved, or whether zero-shot transfer holds on harder or more adversarial agent tasks than ALFWorld and WebShop. Take the specifics as reported, not as settled.

The part worth watching is the direction. If dense signal from a mechanical grader can substitute for hand-designed reward shaping, teams without deep RLHF budgets get a plausible shortcut, and the interesting question shifts from 'where do we find human feedback' to 'where can we find a cheap oracle that grades intermediate states well enough.'