ParaTTT claims 32× faster TTT-MLP eval, 4.9× faster training

ParaTTT, an algorithm from Noel Loo, Neehal Tumma, T. Konstantin Rusch, Mathias Lechner, Alexander Amini and Daniela Rus, reports a 32× speedup over the existing TTT-MLP implementation at 4K context length, with a full 340M model training 4.9× faster. The preprint, posted to arXiv on October 28, targets a known bottleneck in test-time training layers: nonlinear memory networks trained online by gradient descent that the authors say 'run several times slower than chunkwise-parallel linear recurrences' like Gated DeltaNet and Mamba-2.

The paper swaps sequential block-wise weight updates for an iterative refinement loop. Each refinement runs 'three chunkwise-parallel passes: a forward pass that evaluates the resulting feature trajectory, an exact gated-delta-rule pass that computes the readout trajectory and its feature gradients, and a backward pass that propagates these gradients into a new estimate.' The authors prove 'geometric convergence for a two-layer variant under sufficient contraction conditions' and report that in practice 'two iterations suffice.' Two researchers we track flagged the paper in the days after it landed.

At 340M and 1.3B parameters, the authors write that trained models 'match or exceed Gated DeltaNet and Mamba-2 and remain competitive with same-size Transformers on language-modeling perplexity and zero-shot accuracy, with better recall than the linear recurrences.' The abstract publishes no per-benchmark numbers, and no independent replication is attached — the comparisons are the authors' own.