Copy-Paste head reaches LM final loss up to 2× faster at 1.7B

A new transformer tweak reaches the same pretraining loss up to twice as fast as the vanilla baseline, with one caveat up front: the 2× figure is only reported at 1.7B parameters under sliding-window attention. In full and hybrid attention setups the gain is 1.25×.

Nathan Godey and Yoav Artzi call the method Copy-Paste Language Models, or CPLMs. The architecture bolts a second head onto the standard LM head whose job is to point back at a token already in context, gated by a dummy `<copy>` token the LM head learns to emit. "We validate our approach by pretraining models from 135M to 1.7B parameters, showing improvements across all scales and backbones with negligible training and inference overheads," the paper reports.

The authors motivate the design with a statistic from their introduction: roughly 60% of the tokens in FineWeb-Edu already appear earlier in their 4k-token context. Treating copying as a dedicated operation, rather than something a transformer should discover on its own, routes gradient around what the paper calls the LM head bottleneck.

Two other numbers travel with the result. The paper claims an 11.3% improvement over the prior NanoGPT speedrun record, and reports that the 1.7B sliding-window CPLM matches a full-attention LM at 12× less memory per token. All checkpoints are posted at huggingface.co/collections/nthngdy/cplm. The abstract publishes no per-benchmark downstream numbers.