AgentCompass unifies 20+ agent benchmarks under one runtime

For anyone building or evaluating LLM agents, one of the quieter frustrations of the last two years has been how much of the work is plumbing. Every benchmark ships with its own harness, its own environment shims, its own quirks about how to score a trajectory. A group of 23 researchers led by Kai Chen has now put a paper on arXiv proposing something more boring, and more useful, than another leaderboard: a common substrate for the plumbing.

The framework is called AgentCompass, and its central claim is architectural. Instead of treating each benchmark as a monolith, the authors split agent evaluation into three independent components, Benchmark, Harness, and Environment, so pieces can be mixed without reimplementing execution logic. Around that, they wrap a fault-tolerant asynchronous runtime and a set of trajectory-analysis tools the paper says can help diagnose failure modes including reward-hacking.

The reason this matters, if the design holds up, is that it drags the interesting comparison back to the model. A lot of the variance reported between agent scaffolds today is scaffolding variance dressed up as capability variance. A shared harness with shared environments means two teams reporting on the same benchmark are more likely to be reporting on the same thing. The authors claim native support for over 20 benchmarks across five capability dimensions, which if accurate would make this one of the broader open evaluation stacks in circulation.

The honest caveat is that the arXiv abstract doesn't name which twenty-plus benchmarks are actually wired up, doesn't spell out the five capability dimensions, and doesn't give performance numbers. That is the part you would want to see before treating it as a category standard rather than one more framework. And a runtime like this is only as good as the maintainers who keep pace with each new benchmark as it lands.

If it does gain traction, the upside is real. Academic labs and small startups get a common surface that could compress weeks of per-benchmark harness engineering into configuration, and reviewers get a fighting chance at reproducing agent leaderboard claims. The open question is whether the community converges on this substrate, or whether AgentCompass becomes the twenty-first incompatible harness.