SetwiseEvalKit: best RAG reranker tops out at 45% coverage

There is a small result buried in a new arxiv paper that is worth sitting with if you are building RAG for AI agents rather than for humans. The authors argue that the metric everyone actually optimizes for — nDCG-style scoring of documents one at a time — is measuring the wrong thing once the primary consumer of the search results is a model, not a person.

Their framing is that when an agent reads a set of retrieved documents, what matters is how those documents interact: redundancy, conflict, complementarity. Existing evaluation, they say, ignores that entirely and cannot answer what makes one document set better than another. To measure the gap they built SetwiseEvalKit, a three-level, nine-dimension benchmark comprising approximately 28K high-quality evaluation rubrics that cover both short-form and long-form scenarios, and ran twelve rerankers through it.

The headline number is the one to sit with: even the best method achieves no more than 45% coverage, and cross-document coordination dimensions are universally weak. No single method maintains top performance across both settings, which is a quieter but arguably more damaging finding for teams that validate their retrieval stack on one shape of task and ship it against another. Their proposed method, Rubric4Setwise, is training-free and converts rubric-based evaluation criteria into document set selection signals; the paper claims it reaches the best downstream generation performance with fewer documents and search rounds.

The honest caveats matter here. This is a single arxiv preprint, and the 45% coverage figure is measured against a benchmark the same authors designed, so the framing is theirs. What the abstract does not give you is which twelve rerankers were tested, how the individual scores broke down, or how much extra compute the rubric approach costs at query time. Take the specifics as reported, not settled.

The forward-looking part is what a set-level, rubric-oriented lens might unlock. If it catches on, RAG evaluation vendors have a new axis to build against, agent builders have a plausible route to fewer search rounds and lower cost per answer, and enterprises quietly running reranker stacks tuned only on single-document relevance now have a testable reason to look at them again.