ALiBi's FP Underflow Quietly Blinds Long-Context Attention

A new arXiv paper from Christopher Schröder and colleagues argues that ALiBi, one of the more widely used positional encoding schemes for long-context decoder models, has a numerical failure mode hiding in plain sight. The authors show that ALiBi's linear bias scaling underflows floating-point precision at long distances, zeroing out a significant portion of attention weights and leaving the affected heads what they call partially blind.

The reason to chase this down is the asymmetry between what fails and what shows up on your dashboard. According to the abstract, the underflow substantially impairs token retrieval and passkey retrieval, the kind of long-context task teams increasingly rely on, but has only a minor effect on standard decoder benchmarks. A team could ship a long-context model that scores fine on the usual evals and still be quietly worse at pulling a specific fact out of a long document.

The paper proposes four training-time fixes and reports that log-scaled distances give the most consistent improvements in passkey retrieval. The authors still describe default ALiBi slopes as a surprisingly strong baseline, particularly for needle-in-the-haystack retrieval, so the framing is a correction to an existing scheme rather than a call to abandon it.

The honest caveat is scale. The experiments described in what I read were run on 148M-parameter decoder models, and the page does not name specific production models or quantify how much of the deployed long-context ecosystem is affected. What the reporting also does not give you is an inference-time workaround, so teams that have already trained ALiBi models cannot simply patch the bias in place. For anyone starting a new long-context run in the next few months, the cheap move is to add a passkey-style probe to eval and to try a log-scaled variant before committing to the default slopes.