BooStSa automates bootstrap significance testing for NLP models

Statistical significance testing gets skipped a lot in NLP model comparisons, which is why the BooStSa demo from ACL 2022 is worth pulling out of the archive. Tommaso Fornaciari, Alexandra Uma, Massimo Poesio and Dirk Hovy presented the tool in Dublin, and the pitch is straightforward: automate bootstrap-sampling significance tests across many experimental conditions, so researchers stop skipping the step because it is tedious.

The authors' own framing is that 'producing slightly higher performance is insufficient' when you are comparing models. You need evidence the delta will generalize. Bootstrap sampling is one of the more honest ways to get that evidence, but running it by hand across many runs and many conditions is the kind of chore that pushes teams to just report the higher number and move on. BooStSa's contribution is less a new statistical idea than a wrapper that removes the excuse.

What makes it interesting beyond convenience is that it handles both standard categorical predictions, pick a class and count the hits, and soft-label outputs, where the model returns a probability distribution across classes. That second mode matters if you work on tasks with real human disagreement, where the 'right' answer is itself a distribution rather than a single label, and hard-accuracy testing throws away signal about how the model is spreading its probability mass.

The honest caveat is that a bootstrap p-value tests whether an observed difference is likely under resampling of the same evaluation data, not whether the improvement will hold on genuinely new distributions. What the paper does not resolve is the harder question of multiple-testing correction when you are comparing dozens of conditions at once. But for teams still shipping model comparisons with no significance test at all, the tooling excuse is smaller now, which is a boring but useful direction of travel for the field.