LiveMACEBench: live markets expose LLM outcome-capability gap
TL;DR
- Five frontier LLMs traded through 30 days of live markets under matched Tool Use, Persistent Memory, Rule Following, and Multi-Agent Collaboration configurations.
- The authors report that realized returns often diverged from capability-specific measurements, with similar outcomes arising from markedly different mechanism use.
- Trace-level diagnostics show mechanism access, effective mechanism use, and downstream performance are not interchangeable measures of agent capability.
Five frontier LLMs ran 30 days of live-market trading inside a benchmark its authors call LiveMACEBench, and realized returns kept diverging from the capability profiles that produced them. The paper, posted to arXiv, reports a "pronounced outcome-capability gap" across matched configurations for Tool Use, Persistent Memory, Rule Following, and Multi-Agent Collaboration.
The authors write that "realized returns often diverge from capability-specific measurements, and similar outcomes can arise from markedly different patterns of mechanism use." Trace-level diagnostics, built from complete decision logs, further show that "mechanism access, effective mechanism use, and downstream performance are not interchangeable measures of agent capability."
The abstract names neither the five models tested nor any specific return figures. The authors frame the contribution as turning live markets "from a performance leaderboard into a diagnostic environment," aimed at how the field evaluates persistent LLM agents.
Originally reported by paper
Read the original article →Original headline: LiveMACEBench: 30-Day Live-Market Trial Finds LLM Agent Returns Systematically Diverge From Capability Scores