EPSBench: no LLM beats analyst EPS consensus; bands miss
TL;DR
- EPSBench tested six LLM agents on 149 firm-quarters from 86 US companies; none beat the date-matched analyst EPS consensus.
- Stated 80% prediction intervals contained the reported EPS in only 37-70% of runs, with most misses falling above the upper bound.
- The authors trace most of the error gap between the six models to their operating-income estimates.
Across 149 firm-quarters from 86 US companies, none of six LLM agents tested beat the analyst consensus on earnings per share. Three roughly matched it. Three did worse.
That is the headline result from EPSBench, a benchmark posted to arXiv that evaluates LLM agents on quarterly GAAP diluted EPS. Agents read a fixed collection of filings, earnings-call transcripts, and news, then estimate the income-statement drivers themselves; a shared accounting calculator rolls those line items up into EPS so models can be compared on the assumptions that produced the number, not just the number.
The calibration picture is worse than the point-estimate one. The paper reports that "nominal 80% intervals contain reported EPS in only 37--70% of runs that return an interval, and most misses fall above the upper bound because the forecasts run low." The agents are not only overconfident about their own precision; the direction of their miss is predictable too. They lean low.
When the authors look for where models separate from each other, they point at one line of the income statement: "The driver records trace most of the error gap between models to operating-income estimates."
The abstract does not name the six models or publish per-model numbers.
Originally reported by paper
Read the original article →Original headline: First Real-Earnings AI Benchmark: Best LLMs Merely Match Analysts, Intervals Miss by Half