KAIST ACE-RL Trains One LLM in Three Roles, Tops PPO and GRPO
TL;DR
- KAIST AI's ACE-RL co-trains one language model as actor, insight reflector, and playbook curator on top of a fixed Agentic Context Engineering harness.
- The method beats PPO, GRPO, ERL, and end-to-end PPO baselines on held-out agentic-benchmark reward.
- Trained with three attempts per task, ACE-RL still outperforms a GRPO-trained model allowed up to eight attempts.
Co-training a single language model to act, reflect, and curate its own playbook outperforms standard RL recipes on agentic benchmarks, according to a new paper from KAIST AI's Doyoung Kim, Byeongguk Jeon, Kailin Zhang, and Sherry Yang.
Their method, ACE-RL, fixes the Agentic Context Engineering harness in place and grades each candidate experience-handoff artifact by its downstream effect on future rollouts. The abstract reports the model "co-optimizes the role-specific behaviors (action generator, insight reflector, and playbook curator) under a fixed scaffold" and "achieves higher held-out performance than baseline methods" against PPO, GRPO, ERL, and end-to-end PPO.
Gains extend beyond the training cutoff. The paper says that "trained with 3 attempts per task, ACE-RL outperforms GRPO up to 8 attempts", and that training only the actor head is insufficient.
The framing is blunt: "agent training should optimize not only the actions in the current rollout, but also the model-written experience that future rollouts inherit." The release lands amid a steady run of RL-recipe work surfacing on our fine-tuning tracker over the last 90 days.
Originally reported by huggingface.co
Read the original article →Original headline: KAIST's ACE-RL Trains One Model as Actor, Reflector, and Curator, Outperforms PPO and GRPO on Held-Out Reward