Yale-Cardiff Paper Flags 'Derivation Gap' in LLM Tool-Call Judges

LLM judges that score tool-call arguments systematically accept a correct value less often when the input text carries no reference for it, a pattern a new paper from The Fin AI, Yale, and Cardiff University calls the 'derivation gap' and measures across seven judges on three benchmarks.

The gap appears when a value comes from a schema default the user never mentioned, when it has to be inferred, or when the supporting text was cut from the judge's window. On FC-RewardBench defaults, Mistral-7B's acceptance of correct values drops 50.51 points relative to visible-reference cases, and Llama-3.1-8B's drops 49.00 points. On ComplexFuncBench, out-of-view gaps for all seven judges, including the System One scorers CLM and Jev, run between 22.67 and 36.13 points. 'The gap is not only a truncation artifact,' the paper reports, since correct defaults produce the largest gap on a benchmark with no truncation at all. A controlled source-relocation test shows the behavior directly: judges 'favor a visible incorrect value over a correct value whose source is hidden.'

The authors propose DeRef, in which 'a separate model reads the full history without seeing the candidate and writes a reference for the judge.' 'On the controlled test, it raises AUROC from chance or below to between 98.40 and 99.59,' the abstract states. Across all three benchmarks DeRef also beats the RevisEval baseline, with gains such as +6.33 points for CLM on FCRB and +3.77 for CLM on ComplexFuncBench.

The study evaluates seven open models; proprietary frontier judges are not among them. The paper lands amid a steady run of Hugging Face papers in our tracker, 47 in the last 90 days, several on judge and reward-model behavior.