ElephantBench: Best LLM Recalls Both Sides on 52.4% of Questions

The strongest of 32 large language models tested on a new closed-book benchmark recovered both accounts of a disputed long-tail fact only 52.4% of the time. On nearly all of the remaining questions, the model recalled one account and omitted the other.

The benchmark, ElephantBench, is described in an arXiv paper titled "Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge." It comprises "1,094 questions generated through an auditable graph-based pipeline" that pulls related documents from a low-exposure web corpus, identifies naturally occurring disagreements between them, and converts each into a multi-account question-answer record. Every answer, the authors write, "is verified against the originating documents and authoritative public web sources and is then reviewed by human annotators."

More compute is not the fix. "Scaling model size and inference-time reasoning improve recall but do not eliminate this incompleteness," the paper reports. What does correlate with more complete recall, according to the authors' corpus analysis, is training-data exposure: "exposure imbalance favors the dominant account, whereas greater minority-side exposure is associated with more complete recall."

The abstract does not publish per-model numbers or name which of the 32 models sat at the 52.4% top of the pack. Code and data are posted at Tencent's ElephantBench GitHub repository.