StartLux's DecisionPivot Finds Aggregate Scores Hide Gaps

StartLux published DecisionPivot, a 48-task diagnostic benchmark with 40,033 items spanning seven decision-model roles used inside LLM systems: answerability, guardrails, output verification, agent tool checks, reasoning classification, record reconciliation, and constraint-based matching. The headline result from the paper is that aggregate reference scores can be misleading, as "two 27B models separated by only 2.6 points on the reference track achieve contrast-group accuracies of 70.1% and 44.2%."

That gap is the point. On controlled contrast groups, where every member of a paired case must be answered correctly, models with near-identical reference totals diverge sharply. Across 2,400 binary contrast pairs, the authors report Perplexity Decider "correctly ranks the positive member above the negative member in 92.3% of pairs but answers both members correctly in only 74.1%," what they frame as a gap between probability ordering and decision correctness.

Reasoning effort moves the needle. GLM-5.3 climbs from 81.5% pair accuracy at its lowest reasoning setting to 93.5% at a higher one, with average reasoning length rising from 121 to 265 tokens per item. Yet aggregate stability can mask churn: a second run of AutoThink produced 87.2% pair accuracy against the original 86.9%, while 5.6% of individual answers changed underneath the near-identical headline. The authors write that "stable aggregate scores can mask changes in individual predictions."

One caveat the paper flags itself: DecisionPivot and the StartLux-Decision models were developed by the same team, which the limitations section marks as reason that independent evaluation would be valuable. The authors also removed 3,016 reference items, 8.5% of that track, after detecting them in training files before computing the headline rankings.