Tibshirani trio derives optimal score for conformal prediction

The optimal conformal predictor scores candidate labels by the inverse conditional density of Y given X.

That is the concrete finding in a new arxiv paper by Ryan J. Tibshirani (UC Berkeley), Rina Foygel Barber (University of Chicago) and Aaditya Ramdas (Stanford), which reframes conformal prediction as a hypothesis test. Two of the researchers we follow shared the preprint within a day of it going up.

The reframe is small on paper. The authors "again cast conformal prediction via the inversion of a permutation test, but for the null of exchangeability of the joint distribution of the n+1 samples." That switch, from testing a specific candidate y-value to testing whether the whole augmented sample is exchangeable, lines conformal prediction up with hypothesis testing the way classical statisticians already do it.

The payoff is that decades of theory port over. The paper reports that "foundational universality and impossibility results in the conformal prediction literature can be reproduced directly using classical hypothesis testing theory (due to Neyman, Lehmann, Scheffé, Kraft, Le Cam, and others)."

Then Neyman-Pearson gets pushed further. For any joint distribution of the covariates and response and any sample size, the paper says, "the optimal method for prediction sets—which delivers the most efficient set among all methods with valid coverage for exchangeable distributions—is a conformal predictor whose score is the inverse conditional density of Y|X."

The abstract publishes no experiments, no benchmarks, and no guidance on how to estimate that inverse conditional density inside a real model.