ArenaFlow proposes hierarchical credit for open-ended agent RL

ArenaFlow, a preprint posted to arXiv on September 18, proposes to stop compressing an agent's whole trajectory into a single scalar reward and instead run tournaments between trajectories, then propagate credit hierarchically down to individual steps and reusable skills.

The paper frames the problem plainly. Reinforcement learning "has substantially improved large language model (LLM) agents in verifiable domains, but remains difficult to apply to open-ended agent tasks, where solutions are diverse and reliable scalar rewards are hard to obtain." Recent pairwise evaluation methods help, the authors write, but "still compress rich comparative feedback into a single trajectory-level reward, obscuring decisive intermediate steps and preventing successful behaviors from being consolidated into reusable skills."

ArenaFlow's fix uses "tournament-based relative ranking" to score trajectories, and pairs each comparison with a structured reflective evaluation that surfaces "three types of supervision: pivotal success steps, reusable strategy skills, and usage attribution of retrieved skills." At the step level, trajectory-level advantages get pushed onto "high-confidence pivotal steps according to tournament survival depth." At the skill level, the framework "maintains a global skill memory through utility-aware updating, pruning, and retrieval," and the surviving high-utility skills then serve as policy priors for later exploration. Two researchers we track shared the link on submission day, which is how it surfaced here.

The abstract publishes no benchmark numbers, no baseline comparisons, and no named tasks. Only the claim that "extensive experiments validate ArenaFlow's effectiveness on open-ended agent tasks."