RealCLI: top CLI agents score 37.6% on human collaboration
TL;DR
- RealCLI's best agent configuration averaged 49.2% on task execution but peaked at only 37.6% on the joint collaboration score.
- The benchmark pairs 101 tasks across 10 CLI categories and 52 subdomains in two modes: Exec with full spec, Collab with a scripted user.
- Models consistently struggled with steerability, trust calibration, and safety boundary management once collaboration was scored.
The best CLI-agent configuration tested by RealCLI, a new paired benchmark described in a paper submitted to arXiv on October 15, averages 49.2% on task execution and peaks at just 37.6% on the joint collaboration score.
The suite contains 101 tasks across 10 CLI categories and 52 subdomains. Each task has two modes 'sharing the task goal, environment, and verifier': RealCLI-Exec hands the model the complete specification upfront, while RealCLI-Collab 'introduces a scripted user during execution.' On top of the tasks sits a five-dimensional evaluation structure with 22 sub-dimensions and 51 interaction templates.
The authors report that 'task effectiveness diverges sharply from collaboration quality, with models consistently struggling in steerability, trust calibration, and safety boundary management.'
The abstract does not name which frontier models sit behind 'the best configuration,' nor break out which of the 22 sub-dimensions drove the biggest gap between the two modes.
Originally reported by paper
Read the original article →Original headline: RealCLI: Best CLI Agents Score 37.6% on Collaboration Quality vs 49.2% on Task Completion