DiffusionGemma hits 1,500 tokens/sec via parallel refinement

A new technical report posted to arXiv describes DiffusionGemma, an experimental open-weight language model that generates text by refining blocks of 256 tokens in parallel rather than emitting them one at a time. The paper reports about 1,500 output tokens per second on a single NVIDIA H100 GPU, which is a very different neighbourhood from what standard autoregressive decoding reaches on the same hardware.

The mechanism is discrete diffusion. Instead of the usual next-token loop, the model produces around 20 tokens per forward pass and iteratively refines a 256-token block until it converges. The starting point is a mixture-of-experts Gemma 4 with 3.8B activated and 25.2B total parameters, and the team says fine-tuning to the diffusion objective took under 10% of the original autoregressive model's training token budget. Training happens in two stages, supervised fine-tuning for bidirectional denoising followed by reinforcement learning with sampler distillation.

Why this is interesting: for the last couple of years, faster inference on frontier-class models has mostly meant speculative decoding, quantisation, or throwing more silicon at the problem. A parallel-refinement decoder that the authors say sets a new Pareto frontier for the speed-versus-capability trade-off is a different bet, and it comes with the compatibility notes you would want. Thinking mode, multimodal input, and long context all still work, and the model even retains an autoregressive generation mode with what the paper describes as a minor performance loss. Three experts in our Who's Who directory have already circulated the paper, which is unusual for a single-lab benchmark claim.

Every number here comes from the authors themselves, measured on one high-end accelerator. The paper doesn't compare the quality curve to autoregressive Gemma on long-form reasoning, doesn't show how the throughput holds up on smaller or non-Nvidia hardware, and doesn't commit to whether the weights will actually reach the open community, or under what license.

If the 1,500 tok/s claim survives independent testing at real serving batch sizes, the people who benefit first are inference platforms running Gemma-class models for latency-sensitive apps, and the small teams who want capable open-weight inference without a speculative-decoding infra stack behind it.