ScholarCatalyst: Agents Trail Their Own Retriever Tool at 0.42 R@20

An agentic search harness scored 0.42 Recall@20 on a new paper-retrieval benchmark. The embedding retriever it called as a tool scored 0.48 on the same queries. That inversion is the headline finding of ScholarCatalyst, posted to Hugging Face Papers, in which 184 lead authors of 207 recent computer science papers labeled which earlier works "did or could have advanced their project," each with a written rationale.

The task is constrained in a specific way. Given an initial research question, systems may only draw from literature available when the project began. No peeking at the citations the finished paper eventually made. Even an agent built on Claude Fable 5.1, which the authors note "may have seen the completed papers during training, reaches only 0.51 R@20."

"What makes great scientists great?" the abstract opens, answering that scientists "remain far ahead" of AI systems at "sensing which prior idea, buried in an ever-growing archive of research, a new problem needs." The paper argues the results "highlight the need for new training recipes that equip models with expert intuition for searching broad corpora."

It lands in a modest run of agent-retrieval papers our tracker has been catching; a separate benchmark paper this week argued that lenient scoring was crediting tool mentions as actual tool calls. ScholarCatalyst's agents do call the tool. They just underperform it.