Shanghai AI Lab's S3AC Trains Working Critic for Long-CoT RL at 397B

Shanghai AI Laboratory researchers say they have trained a working critic for long chain-of-thought reinforcement learning at scales from 1.5B to 397B parameters, in an area where critic-free methods currently dominate because training critics has been too costly. The method, called S3AC, is laid out in a paper posted to arxiv by Shuo Shen, Guangran Cheng, Chengqi Lyu, Xiangsen Wang, Yanan Sun and Wenwei Zhang.

Critic-free approaches like GRPO took over RLVR for reasoning precisely because, as the authors put it, "long-CoT critic training is hindered by insufficient coverage of trajectories with diverse lengths and early gradients dominated by the newly initialized value head." S3AC is pitched as a practical recipe that works around both failure modes.

A trained critic gives two things group-rollout baselines lack: "token-specific advantages and prompt-specific baselines." The second feeds a sampler the authors call Difficulty-Aware Prompt Sampling, which uses the critic's own baseline estimate, rather than any pre-labeled difficulty set, to drop prompts that would waste rollouts before generation happens.

The headline claim is scaling. S3AC improves group-rollout performance across dense and mixture-of-experts models from 1.5B to 397B parameters, "outperforming a modern GRPO variant (GRPO-X)," and at 35B parameters it "sustains single-rollout training while tested critic-free baselines deteriorate after initial gains." The abstract names no base models, benchmarks or per-task numbers.

That gap matters alongside last week's HKUST finding that reasoning habits trace to pretraining data rather than RL: if critic-free setups are also leaving credit assignment on the table, the design space for RL-for-reasoning has more room than the field's current recipes suggest.