NVFP4 Needs Up to 1.52x More Exposures Than BF16 for Rare Facts

NVFP4 pretraining takes 1.32x to 1.52x as many exposures as BF16 to match rare-fact recall even when validation loss differs by only 0.5% to 2%, according to a new arXiv paper by Chang Lu and Mengjiao Zhang. The pair trained transformers from scratch with emulated 4-bit matrix multiplications on synthetic biographies with controlled exposure counts, run both standalone and mixed into web text.

The paper's framing is that loss parity is not fact parity. "NVFP4 needs 1.32× and 1.52× as many exposures as BF16 to match its recall while validation loss differs by only 0.5% and 2%," the authors write, and when both models are evaluated with BF16 matmuls at inference the ratios are 1.26x and 1.30x. Under what they call their primary readout, "the exposure cost is about 1.2× at matched loss."

Lu and Zhang then isolate where the cost comes from. "Zero-mean gradient noise reproduces the cost of quantizing only the backward pass, which barely moves the loss." In a controlled experiment at twice NVFP4's error, noise injected only into Adam's second-moment estimate behaves like a lower learning rate and fades out, while noise in the update itself carries about two thirds of the cost through the end of training. Raising the learning rate to offset the smaller average step does not close the gap.

Their closing recommendation is direct: "We suggest measuring rare-fact acquisition alongside loss when evaluating low-precision recipes." The paper lands into a thick week of training-method research on AI Weekly's research tracker, which has logged 52 stories in the last 90 days.