ProbabilityBench measures model calibration via exact Bayes error

ProbabilityBench tests language models on scenarios where the correct probability for each case is known analytically, not estimated. Across three simulated text-based domains (peer review, covert information transfer, and credit risk), authors Yuyang Dai, Yuting Tang, Futoshi Futami and Takashi Ishida define the full data-generating process, then "analytically compute both exact class probabilities and Bayes error" for every instance.

Standard calibration evaluations have to approximate the right answer because real-world data gives only one observed label per case. ProbabilityBench sidesteps that by rendering each simulated case into a natural-language memo that, the paper argues, "strictly preserves their underlying generative probabilities," so a model's reported probability can be scored directly against truth via KL divergence.

Across 16 evaluated models, "most models on the observed cost-quality Pareto frontiers are decision models, including Jev and Perplexity Decider." The abstract reports no per-model scores beyond that Pareto note. The authors also position the benchmark as "a meta-evaluation framework," using the known ground truth to grade standard calibration metrics and Bayes-error estimators themselves.