JudgmentBench: pairwise scoring beats rubrics 0.908 to 0.150

On 30 real-world legal tasks graded by practicing attorneys, pairwise preference judging of LLM outputs reached a Spearman rank correlation of 0.908 with the intended quality ordering. Rubric scoring, on the same tasks and the same outputs, reached 0.150.

That is the headline from JudgmentBench, a paper first posted to arxiv in May and revised on November 5, listing NeurIPS 2026's Evaluations and Datasets Track as its venue. The authors, a team that includes Julian Nyarko, collected 1,539 rubric scores and 1,530 pairwise preference judgments from attorneys at major U.S. law firms, grading LLM-generated outputs at three quality levels. The second metric moves the same direction: pairwise win-rate comparison hit 0.669 against 0.542 for rubrics, an estimated gap of 0.127.

The paper's own line is blunter than the numbers suggest: "comparative judgments recover the intended quality ordering substantially better than rubrics." It also reports that preference-based evaluation required "less than half the annotation time" of the rubric approach, and that the pattern holds across both human and LLM evaluators.

Two researchers we track posted the paper's link in the days after the November revision landed.

The dataset is narrow (30 tasks, three quality levels, lawyer annotators), and the summary does not break out how much of the gap is rubric-design-dependent versus inherent to the format outside legal work.