Devezer & Buzbas: replicability itself isn't replicable

Two University of Idaho statisticians argue that the standard method for declaring a replication crisis in science cannot itself establish that a crisis exists. In a preprint posted to arXiv on 29 April 2026, Berna Devezer and Erkan O. Buzbas formalise two ways to interpret replication studies and show that neither supports the binary 'replicable / not replicable' label the field has been using.

Under what they call the benchmark model, in which experiments share a common random replicability rate, they write that 'even small variability among replicability rates induces an irreducible variance floor on the estimated mean replicability rate that no amount of replication can eliminate.' Under the operational model, where each experiment has its own rate drawn from a population, they argue that 'one binary verdict per experiment carries no information about between-experiment heterogeneity,' so the degree of non-exactness is not identifiable from standard replication data.

The practical consequence, they write, is that 'the usual data structure cannot support reliable demarcation between "replicable" and "not replicable" results and systematically understates uncertainty, making high- and low-replicability sequences appear discriminable when they are not.' Aggregating rates across heterogeneous literatures, they add, 'produces averages that conflate incommensurable regimes and lack a stable interpretation.' The paper demonstrates the effect through a reanalysis of Many Labs 4.

Two researchers we follow in the metascience space circulated the preprint shortly after it went up.

The abstract's closing line states the position flatly: 'The replication crisis, if there is one, cannot be established by the methods used to declare it.'