blog.google via Reddit

Google DiffusionGemma delivers 4x faster text gen

google open source inference ai-models open-source inference

Key insights

  • DiffusionGemma generates 256 tokens per forward pass using bidirectional attention, reaching 1,000+ tokens/sec on a single H100 GPU.
  • With only 3.8B active parameters during inference and an 18GB VRAM footprint when quantized, it runs on consumer hardware without server-grade resources.
  • Google recommends DiffusionGemma only for speed-critical workloads like in-line editing and code infilling, not for applications requiring maximum quality.

Why this matters

DiffusionGemma demonstrates that discrete diffusion text generation is now deployable at the open-weight tier, with integrations across production inference stacks including vLLM, NVIDIA NeMo, Google Cloud's Model Garden, and NVIDIA NIM. The combination of 1,000+ tokens/sec throughput on an H100 and a quantized 18GB VRAM footprint changes the cost calculus for real-time code infilling products that currently require dedicated inference clusters. Apache 2.0 licensing through Hugging Face compresses the timeline for practitioners to benchmark diffusion against autoregressive baselines directly in their own pipelines, without waiting for commercial APIs.

Summary

Google released DiffusionGemma on June 10, a 26B MoE model generating text via diffusion rather than sequential token prediction. The model starts with "a canvas of random placeholder tokens" and makes "multiple passes, locking in correct tokens and using them as context clues to refine the rest." The benchmark: 1,000+ tokens/sec on a single H100, 700+ on an NVIDIA GeForce RTX 5090, generating 256 tokens per forward pass. Quantized to 18GB VRAM with only 3.8B active parameters during inference, it runs on consumer hardware. Essentially: Research Scientists Brendan O'Donoghue and Sebastian Flennerhag applied Gemini Diffusion research to the Gemma 4 architecture, releasing it under Apache 2.0. - Targets speed-critical, interactive local workflows including in-line editing, code infilling, markdown formatting, and amino acid sequence generation. - Google explicitly recommends standard Gemma 4 for applications requiring maximum quality. - Available via Hugging Face, vLLM, NVIDIA NeMo, Google Cloud's Model Garden, and NVIDIA NIM. Speed is the argument; the quality gap is the only test that matters for production adoption.

Potential risks and opportunities

Risks

  • If the quality gap versus standard Gemma 4 proves large on production benchmarks, teams that adopted DiffusionGemma for code infilling pipelines may face costly rollbacks within the first 60-90 days of deployment.
  • Consumer GPU users targeting 700+ tokens/sec on the NVIDIA GeForce RTX 5090 may find the 18GB VRAM quantization constraint conflicts with multi-model or multi-task serving setups, limiting practical deployment flexibility.
  • Enterprises integrating via Google Cloud's Model Garden or NVIDIA NIM inherit those platforms' pricing and availability constraints, creating a dependency on third-party infrastructure that the Apache 2.0 license does not eliminate.

Opportunities

  • Inference optimization vendors already listed as integration partners, including vLLM, Unsloth, and NVIDIA NeMo, can capture developer mindshare early by shipping optimized DiffusionGemma support before competing runtimes finalize it.
  • Real-time code editor products such as those targeting in-line editing and code infilling can make a concrete architectural argument for diffusion-based generation at 1,000+ tokens/sec on H100 hardware, targeting sub-second latency for multi-line completions.
  • NVIDIA benefits as DiffusionGemma shifts the decode bottleneck from memory-bandwidth to compute, directly validating the H100 and GeForce RTX 5090 as the preferred hardware for diffusion inference workloads at both the enterprise and consumer tier.

What we don't know yet

  • Quality gap versus standard Gemma 4 is acknowledged but not quantified: no benchmark scores such as MMLU or HumanEval are provided to let practitioners calibrate the actual tradeoff.
  • How latency compounds across multiple 256-token forward passes for longer completions is unaddressed, making it difficult to translate the per-pass throughput numbers into realistic end-to-end response times.
  • Whether the amino acid sequence and markdown formatting use cases extend to other structured non-natural-language domains (SQL, regex, structured data) is not addressed, leaving the scope of non-linear generation unclear.

Shared on Bluesky by 4 AI experts

Originally reported by blog.google

Read the original article →

Original headline: Google DeepMind Releases DiffusionGemma — 26B MoE Open-Weight Model Generates Text 4× Faster via Diffusion, 1,000 Tokens/Sec on H100