'Full-bandwidth transformer' claims 1.5x token efficiency

A new preprint from Xi Wang and coauthors proposes a small structural change to autoregressive transformers that they claim recovers a meaningful chunk of the value normally bought by training on more tokens. The idea, laid out in the arxiv paper, is that today's decoding feeds only the sampled token back into the bottom of the stack while discarding the top-layer hidden state. Their fix, which they call latent feedback, fuses that hidden state with the next token's embedding through a gated linear unit before it re-enters the model.

The reported results are the interesting part. Training 1B-parameter models on up to 400B tokens, the authors say latent feedback improves validation loss, 5-shot language-model evaluation, and math and coding generation, and produces shorter reasoning traces at equal or better accuracy. Their headline claim is that these models match or approach standard transformers trained with roughly 1.5x more tokens, at what they describe as negligible per-token decoding overhead. Training itself uses a scheduled multi-pass objective that introduces latent feedback late in pretraining so that teacher forcing can stay parallel.

The number that carries the weight here is the 1.5x. If a token-efficiency win of that size survives at larger scale, it maps directly onto the compute and dataset budgets that dominate frontier training runs. The paper is careful to note the modification does not break the transformer contract: the KV cache, architecture, and language-modeling objective are preserved, so an inference stack does not need to change to try it.

Results are reported by the authors at 1B parameters and up to 400B tokens, well short of frontier scale, and the 1.5x figure comes only from the paper itself. The submission is recent enough, dated August 9, that no external group has replicated it yet, and the abstract does not spell out how the multi-pass training objective scales in wall-clock cost. Two AI experts we track were among the first to circulate the link, which puts this in the read-and-poke-at bucket rather than the settled-result one.

If the effect survives at larger scale, the beneficiaries are anyone paying the token bill for pretraining; the loser, quietly, is the assumption that dense decoder-only stacks have already been squeezed for architectural free lunches.