JEV Judge Model Matches LLM Accuracy at 0.36% of the Cost
TL;DR
- JEV lands within three percentage points of a frontier LLM judge on preference and factuality tasks at 0.36% of the comparator's fee.
- A cascade that accepts confident verdicts and escalates uncertain ones retains 99% of the comparator's accuracy, per the abstract.
- The gap widens on reasoning-heavy benchmarks: JEV trails by 14.6 points on JudgeBench, where verdicts require checking a derivation.
A judge model that never generates a single token of text lands within three percentage points of a frontier LLM on ordinary preference and factuality evaluations at 0.36% of the price, according to a new paper on Hugging Face.
"JEV-as-a-Judge: Accept When Confident, Escalate When Unsure," by Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman, compares the small decision model against sixteen generative and reward-model judges using blinded human adjudication. On RewardBench, JEV scores 92.2% against the comparator's 93.5%; on HaluEval it edges the larger model at 87.5% to 86.7%.
The gap widens on reasoning-heavy work. JEV trails by 14.6 points on JudgeBench. "Larger gaps arise when judgments require checking a derivation or resisting an elaborately written wrong answer," the authors write.
Their fix is a cascade: accept the small model's verdict when it is confident, forward the rest to the expensive one. That configuration "retains 99% of the comparator's accuracy at lower cost," per the abstract. JEV itself runs on a ModernBERT-large backbone with a small decision head, priced at roughly $0.044 per 1,000 judgments against the comparator's $12.182.
It fits a run of eval-and-oversight work our tracker has logged across the safety beat over the last quarter, where the practical question is increasingly cost-per-judgment rather than raw agreement.
Originally reported by huggingface.co
Read the original article →Original headline: JEV-as-a-Judge Paper Adds Confidence-Gated Escalation to Typed Decision Models