Abra study: compute-optimal diffusion needs 200 tokens/param
TL;DR
- The Abra paper pegs compute-optimal text-to-image diffusion training at about 200 image tokens per parameter, roughly ten times the Chinchilla ratio for LLMs.
- Scaling laws hold across three orders of magnitude of compute, from 10^19 to 10^22 FLOPs, on the authors' Abra flow-matching transformer family.
- Unlike LLMs, diffusion models tolerate overtraining, so the authors recommend prioritizing larger datasets over bigger models.
An Abra team's new scaling study pegs compute-optimal training for text-to-image diffusion at roughly 200 image tokens per parameter, about ten times the data-per-parameter ratio Chinchilla established for language models.
The paper, posted to arXiv on August 18 by Kyle Chickering, Wei-An Lin, Jiaming Song and colleagues, uses a family of flow-matching transformers called Abra to sweep three orders of magnitude of compute, from 10^19 to 10^22 FLOPs. Within that range, the authors write, "compute optimality occurs at approximately 200 image tokens per parameter." Their guidance is to spend surplus budget on more data before scaling the model.
The second finding cuts against the LLM playbook in a different way. Where over-training a language model past its Chinchilla point yields diminishing returns, the Abra results say diffusion models are robust to overtraining. Training curves and downstream measures track together: "this predictability extends beyond training loss to generative quality metrics, optimal CFG settings, representation quality, and even the shape of the training curves, which collapse onto a universal form."
The scope is bounded. The compute sweep tops out at 10^22 FLOPs, well short of frontier image-model runs, and the fit is measured on the authors' own Abra architecture rather than cross-checked against other diffusion families.
Originally reported by paper
Read the original article →Original headline: Abra Study Finds Compute-Optimal Diffusion Training Needs 200 Image Tokens per Parameter — 10× LLMs