Vehtari team benchmarks 11 shortcuts for Bayesian LOGO-CV

Leave-one-group-out cross-validation, a workhorse for judging how a hierarchical Bayesian model will do on unseen groups, normally forces one full MCMC refit per held-out group. A new arxiv preprint from Anna Elisabeth Riha, Svenja Jedhoff, Paul-Christian Bürkner and Aki Vehtari benchmarks 11 shortcuts, five of them newly proposed, for skipping that cost.

The paper frames the job as two coupled tasks: obtaining draws from leave-one-group-out posteriors, and integrating out group-specific parameters. Its headline claim is that 'marginalising over the group-specific parameters substantially improves the reliability of the importance sampling approaches.' Standard PSIS on its own, the authors report, produces systematically biased estimates across their experiments; Laplace approximation paired with PSIS lands as the accuracy-versus-cost sweet spot, while bridge sampling is more accurate but requires substantially more computation.

The methods are pitched to slot into existing Bayesian workflows through the loo, brms, Stan and aghq packages. Two researchers we track posted the link within a day of it appearing.