Sun et al.: LLM agents accurate on tasks, poor at flagging doubt

Higher task accuracy in LLM agents does not translate into greater willingness to flag uncertainty, according to a new preprint, Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict, by Kaiser Sun, Bernal Jimenez Gutierrez, Hongjun Liu, Jingyu Zhang, Jie Gao, Mark Dredze and Daniel Khashabi.

The authors propose evaluating agents on what they call epistemic humility, defined in the abstract as "the agent's willingness to recognize, act on, and communicate uncertainty during task execution." They operationalize it through three trajectory-level behaviors — Identify, Solve, and Escalate (ISE) — and test them against knowledge conflicts, where either the backbone model's parametric priors clash with retrieved evidence or two contextual sources disagree.

Across four agents, the finding is blunt: "higher task accuracy does not necessarily correspond to greater epistemic humility." Some high-accuracy configurations register the conflict mid-run and still deliver a confident wrong final answer. Trajectory analysis adds a second wrinkle. "Agents frequently detect conflicts in early steps of execution but fail to maintain or resolve them in later steps," the authors write.

Fixes come with a bill. The paper reports that "model-level interventions can improve EH, but often at the cost of task accuracy," which the authors read as evidence that epistemic humility "emerges from the interaction among the backbone model, agent harness, and evaluation environment." Two of the researchers we track in our Who's Who directory had circulated the paper by the time we picked it up.

The abstract does not name which four agents were tested, nor the size of the accuracy drop that came with the EH-improving interventions.