BI-Bench: Frontier LLMs Score Under 50% on End-to-End BI

Frontier large language models score less than 50% accuracy on BI-Bench, a new benchmark that asks the model to answer a business question end-to-end from raw Power BI files instead of a hand-prepped table. State-of-the-art NL2SQL systems do considerably worse: 6.0 to 17.3 percent.

That is the top-line result of BI-Agent and BI-Bench: Towards Automating End-to-End Business Intelligence, by Chuxuan Hu, Yeye He, Penny Zhou, Wee Hyong Tok, Daniel Kang and Surajit Chaudhuri. The authors crawled "over 3K real Power BI project files (.pbix files)" and hand-extracted 100 (analytical query, ground-truth table) pairs from the dashboards inside them. They report the labeling consumed "over 400 person-hours, or over 4 person-hours per query."

The framing is deliberately end-to-end. In a production BI workflow, the paper notes, users must "(1) identify relevant tables, (2) perform data transformations, and (3) build join relationships, before they can (4) answer their business questions." BI-Bench refuses to hand any of those steps to the model.

Layering a tool-augmented agent on top closes a lot of the gap, in the authors' account. BI-Agent decomposes the workflow into search, join and transform subtasks and, per the abstract, "achieves substantial accuracy gains of up to 40 percentage points on BI-Bench with vanilla LLMs, and post-trained BI-Agent yields gains of up to 30 points, both of which are highly statistically significant."

The cost figure is the arresting one. A post-trained Qwen3-8B, an 8-billion-parameter open model, runs the entire benchmark for "only $0.19" versus "over $10" for large frontier models, which the paper calls "54× more cost-effective." Total training cost is reported as "less than $200."

The abstract publishes no per-model accuracy table, and the 50%, 40-point and 30-point figures are stated as ceilings ("less than", "up to"), not averages.