Wolf et al. Show LLMs Systematically Violate Total Probability

There is a small but pointed result in a new paper that says something uncomfortable about how large language models handle populations. If you ask a frontier model the same question two ways, once about the whole population, and once by breaking the population into subgroups and adding up the answers, the model gives you two different answers. The authors call the second version, the one built from finer-grained subgroup prompts, more faithful to actual human reference data than the direct top-level answer. They name the gap the macro fallacy.

The setup, laid out in the paper on Hugging Face by Patrik Wolf, Thomas Kleine Buening, Andreas Krause and Celestine Mendler-Dünner, is a binary tree. They recursively partition a population into increasingly fine-grained subpopulations, prompt the model for each slice, and aggregate the pieces back into a population-level estimate. Under the law of total probability, prior-weighted conditional distributions should recompose into the population marginal. In LLMs, they do not. The authors report widespread violations of basic consistency properties across problem domains and state-of-the-art frontier models.

Why this matters if you are not writing a research paper: quite a lot of production use of LLMs quietly relies on this exact pattern working. Synthetic persona surveys, market sizing prompts, and 'what would a typical customer segment say' workflows all treat the model as a conditional estimator over some slice of humanity. If asking about the whole population gives you one answer and adding up the parts gives you a better one, the way you prompt is silently deciding which reality you get. The authors' own framing is that models possess relevant subpopulation knowledge but do not reliably propagate it into aggregate estimates.

The honest caveat is that the writeup does not name the specific frontier models tested, does not put a number on how large the gap is on any given task, and does not say whether the implicit prompting recovery generalizes beyond persona prompting into reasoning or forecasting. Take the direction as the interesting claim, and wait for follow-ups on magnitude.

The forward-looking piece is the framing the authors close on. If statistical self-consistency stays unsaturated as models scale, that is, if bigger and more capable models keep failing this check, then a reference-free criterion that requires no human labels and cannot be gamed by memorizing a benchmark is a genuinely useful new way to track progress rather than leaderboard position.