Q2D-Web pits 13 retrievers on 190M-doc agentic RAG corpus

Most retrieval benchmarks assume a human types the query. Q2D-Web assumes an agent rewrites it first, and then measures what happens. The paper on arXiv pairs 190 million web documents with 70,000 agentic search queries across 10 languages, all reformulated from real user queries, and scores 13 retrievers spanning lexical, dense, and late-interaction families against three separate sets of relevance judgments: agent citations, production rankings, and a combined set with LLM-based judgments.

The headline finding is that the leaderboards agree with each other but not with themselves once you slice the data. The abstract reports that retriever rankings remain relatively stable across the three judgment sets but "vary significantly by topical domain, query language, and query type," a warning shot for anyone who benchmarked in English on one vertical and shipped a multilingual agent. The authors also flag a practical shortcut for teams running the benchmark: retaining one-third of the corpus preserves the full-corpus model ordering, with "Recall@1000 only by 3 to 7 points" difference.

A public leaderboard is hosted on Hugging Face Spaces.