LLM verifiers grow more lenient after audit-repair in context

A completed audit-to-repair episode sitting in a language model's context window makes that same model, acting as a verifier, less likely to flag correct work as wrong. In a preprint on arXiv, Parsa Mazaheri and Kasra Mazaheri report false-alarm reductions of "2.8 to 11.5 percentage points against a length-matched non-audit control, a 9 to 25% reduction relative to that control" across 15 of 15 model-by-wording combinations tested on human-verified-correct ProcessBench traces.

The finding cuts against prior expectations. "The direction contradicts what the accumulated-message literature predicts," the authors write; an episode whose audit reported an error lowers false alarms further still, "at all five wordings on the model where that manipulation lands cleanly," even though "a negativity asymmetry predicts more flagging."

Signal-detection analysis places the shift in the decision threshold rather than in discrimination: "the criterion moves in 15 of 15 combinations and survives correction in 13 while d' survives in none," though the paper flags that "the d' test is half as sensitive by construction."

Whether the leniency is harmful depends on what the flagged items actually were. A hand audit of 50 false alarms found "82% simply wrong," and the paper argues "at this operating point the shift need not be harmful." The effect also keeps its relative size with reasoning enabled on both models tested.