RL expands LLM agent capability on compositional tool use: study

On static reasoning benchmarks, reinforcement learning has not been shown to teach a base model anything it could not already produce with enough samples; the pass@k curves converge. A new preprint from Zhiyuan Zhai and co-authors argues that the picture flips once agents take multiple tool-use rounds. In "Does RL Expand the Capability Boundary of LLM Agents? A PASS@(k,T) Analysis", the authors propose a two-dimensional metric that varies sampling budget k alongside interaction depth T, and report that on compositional tasks "the RL agent's pass-curve pulls above the base model's and the gap widens at large k rather than converging."

The expansion, they write, "is specific to compositional, sequential information gathering; on simpler tasks RL behaves as prior work predicts." That split lets them position the result as reconciling the field's two camps: "both are correct, on different task types." Two of the researchers on our tracker list have shared the paper.

To isolate what is doing the work, the authors run a matched-data supervised fine-tuning comparison. SFT, they report, "regresses the boundary on the same compositional tasks," which they read as evidence that "self-directed exploration" is the causal factor behind RL's gains rather than the training data itself. Their mechanism analysis attributes the improvement to how the RL agent reweights base strategies toward "the subset whose downstream reasoning more often yields a correct answer," with the biggest effect on how the agent integrates retrieved information.