Claim Checklist cuts Gemini agent false reversals to 0.02%

A training-free prompt that has an LLM agent record its own confidence before hearing user criticism cut false reversals of correct work from 12.2% to 0.02% on pooled software-engineering benchmarks, according to a new arXiv paper dated October 27. The method, called Thinking-in-Bets, has the agent write a short private Claim Checklist listing each claim, the evidence behind it, and a confidence score before the user replies. On revision, claims scored at or above 0.5 are kept; weaker ones are fixed.

The authors tested five Gemini models, from 3.5 Flash-Lite through 4.0 Argon, across eight benchmarks spanning coding, finance, and academic reasoning. On Agents' Last Exam, false reversals under mistaken criticism fell from 93.3% to 6.7% on 3.5 Flash-Lite, and post-review pass rate climbed from 30.0% to 95.0%. In interactive coding on SWE-Interact, unnecessary edits dropped from 66.7% to 27.8% on 3.8 Flash while task pass rose from 55% to 70%.

Timing is the point the paper insists on. "Recording claims before the complaint cuts false reversals on passing drafts more than fourfold and raises post-review pass rate by +31 points; writing the same record after reading the criticism recovers very little of the gain," the authors write. Because confidence is attached to individual steps, "the agent can hold firm on steps it verified while still accepting hints on steps it marked as uncertain."

The paper tests Gemini only, uses simulated users rather than real reviewers, and sets its 0.5 confidence threshold by hand rather than tuning it on data. It lands amid a run of agent-reliability work in our tracker, including a JHU-NYU benchmark earlier this month showing top agents flag conflicts and then hide them.