NextLat auxiliary loss pushes transformers toward belief states
TL;DR
- NextLat adds a self-supervised auxiliary loss that trains a transformer to predict its own next latent state given the next token.
- The authors argue these latents provably converge toward belief states, a compact summary of history needed to predict the future.
- Reported gains span world modeling, reasoning, planning and language modeling, with inference sped up by up to 3.3x via self-speculative decoding.
Transformers store history through attention over past tokens, not through a compact internal state. A team led by Jayden Teoh, with coauthors including John Langford and Alex Lamb, proposes a small auxiliary training objective meant to push the model to build one anyway.
The method, Next-Latent Prediction or NextLat, trains a transformer to predict its own next latent state given the next token, alongside standard next-token loss. From the paper's abstract: "this simple auxiliary objective injects a recurrent inductive bias into transformers while leaving their architecture, parallel training efficiency, and inference unchanged." The authors argue the resulting latents "provably converge towards belief states, compressed information about the history necessary to predict the future."
The one hard number in the abstract is a speedup: variable-length self-speculative decoding accelerates inference "by up to 3.3x in language modeling." Gains are also reported "across benchmarks in world modeling, reasoning, planning, and language modeling," but the abstract publishes no per-task figures. Code is posted on GitHub.
Two researchers we track shared the source link into our directory.
Shared on Bluesky by 2 AI experts
-
Christian Wolf @chriswolfvision.bsky.social: Another relevant paper @phillipisola.bsky.social shared at a different place: arxiv.org/abs/2511.05963 →
Originally reported by arxiv.org
Read the original article →Original headline: Next-Latent Prediction Transformers Learn Compact World Models