DiffusionGemma hits 1,500 tokens/sec via parallel refinement

A new technical report posted to arXiv describes DiffusionGemma, an experimental open-weight language model that generates text by refining blocks of 256 tokens in parallel rather than emitting them one at a time. The paper reports about 1,500 output tokens per second on a single NVIDIA H100 GPU, which is a very different neighbourhood from what standard autoregressive decoding reaches on the same hardware.

The mechanism is discrete diffusion. Instead of the usual next-token loop, the model produces around 20 tokens per forward pass and iteratively refines a 256-token block until it converges. The starting point is a mixture-of-experts Gemma 4 with 3.8B activated and 25.2B total parameters, and the team says fine-tuning to the diffusion objective took under 10% of the original autoregressive model's training token budget. Training happens in two stages, supervised fine-tuning for bidirectional denoising followed by reinforcement learning with sampler distillation.

Why this is interesting: for the last couple of years, faster inference on frontier-class models has mostly meant speculative decoding, quantisation, or throwing more silicon at the problem. A parallel-refinement decoder that the authors say sets a new Pareto frontier for the speed-versus-capability trade-off is a different bet, and it comes with the compatibility notes you would want. Thinking mode, multimodal input, and long context all still work, and the model even retains an autoregressive generation mode with what the paper describes as a minor performance loss.

The honest caveat is that all of this is the team's own report, benchmarked on a single high-end accelerator. What the reporting doesn't give you is how the quality curve compares to autoregressive Gemma on long-form reasoning, how the numbers hold up on smaller or non-Nvidia hardware, or whether the weights are actually going to land in the open community, and on what license.

If the 1,500 tok/s claim survives independent testing at real serving batch sizes, the people who benefit first are inference platforms running Gemma-class models for latency-sensitive apps, and the small teams who want capable open-weight inference without a speculative-decoding infra stack behind it.