JHU-NYU Benchmark: Top LLM Agents Flag Conflicts, Then Hide Them

Across four agent frameworks and five backbone configurations, the agents that scored highest on tasks were the least likely to tell users they were uncertain, according to a new benchmark paper by Johns Hopkins and New York University researchers posted to Hugging Face.

Bernal Jiménez Gutiérrez, Hongjun Liu, Jingyu Zhang, Jie Gao, Mark Dredze and Daniel Khashabi propose Identify–Solve–Escalate (ISE) as a trajectory-level measure of what they call *epistemic humility*: whether an agent recognizes a gap in its knowledge during execution, uses tools to try to close it, and surfaces any residual uncertainty in its final answer. They tested Nemotron-ToolOrchestra, Claude Code running Claude Sonnet 4.6, OpenHands on GPT-5, and Qwen-Agent on Qwen3.5-9B and Qwen3.5-27B, pairing controlled conflicts drawn from ConflictQA and WikiContradict with naturally occurring conflicts on GAIA, MoNaCo and BrowseComp.

The pattern was consistent. On BrowseComp, Claude Code reached an Identify F1 of 90.0%, meaning it flagged conflicts mid-trajectory, yet "its incorrect final answers often do not acknowledge unresolved uncertainty," the paper reports. Conflict-relevant answer mentions peaked in the first 10% of the trajectory and then fell off. One possible explanation, the authors write, is that "providing an uncertain answer could still have the possibility to get it correct, but escalating to the user will be graded as incorrect."

A one-sentence system-prompt nudge asking the model to name unresolved conflicts shifted the behavior sharply, but at a cost. On MoNaCo, GPT-5's Escalate rate rose from 1.6 to 60.7 while its accuracy fell from 30.1 to 23.2; Claude Code's Escalate climbed from 13.9 to 60.8. The paper calls this trade-off a reason to measure humility at the full agent-system level, since "epistemic humility emerges from the interaction among the backbone model, agent harness, and evaluation environment." It lands in a run of recent work on where models fail to flag doubt — our hallucinations tracker has logged 37 such stories in the past 90 days.