BM25 Beats Dense Retrieval and Agents by ~20 Points at Scale

A new scaling study on arxiv does something that RAG benchmarks almost never do: it holds the questions and the reader model fixed, and just varies how big the corpus is. Twenty-eight strictly nested tiers, roughly 450-fold in range, one judging protocol, and a fixed bedrock of relevant and adversarial documents. Then it lets lexical retrieval, dense retrieval, graph-based indexing, and an agentic file-system searcher fight it out on the same ground.

The headline finding is a crossover, not a coronation. At the smallest shared tiers the File-System Agent actually wins, which lines up with the intuition that a reasoning loop can beat plain keyword search when the haystack is small. The catch is that the agent's sequential exploration costs 39 times more query tokens at the bedrock, and its accuracy erodes as the search space grows. Around 10 million corpus tokens BM25 overtakes it, and from that point on BM25 leads every larger tier, with a margin the authors describe as approaching 20 points at full scale. Dense retrieval stays efficient but less accurate; graph-based RAG hits construction walls before it even reaches deployment scale.

Why this matters if you are not writing retrieval papers: a lot of production RAG spend right now is going into vector databases, embedding pipelines, and agent frameworks that assume BM25 is the thing you replace. The authors' framing is close to the opposite. They call lexical retrieval the strongest scalable default and put BM25 on the low-cost end of the Pareto frontier because it needs no LLM-based construction step. Agentic reasoning, in their telling, works best after ranked discovery rather than in place of it, which is a very different architectural picture from an agent that browses your files.

The honest caveat is that this is one paper, one reader model, one judging protocol, and the abstract reports scale in corpus tokens rather than in raw document counts, so mapping the crossover to your own corpus takes some work. What the reporting does not give you is a head-to-head against the obvious hybrid setups, BM25 fused with a dense re-ranker or with an agent that only ranks the top candidates. If those numbers land the same way in follow-up work, the practical takeaway is unglamorous but useful: keep the boring lexical baseline in production, and spend the saved tokens on the reader and the re-ranker rather than on replacing the retriever.