Yandex Research Opens FLAMIA Attention Kernels for Blackwell

FLAMIA, a family of attention kernels for NVIDIA Blackwell GPUs, cuts the combined forward and backward attention call by 1.10 to 1.40 times on B200 and 1.06 to 1.42 times on B300, according to a paper posted on Hugging Face on November 3 by Fedor Velikonivtsev, Mikhail Koltakov, and Denis Kuznedelev. The NVFP4 backward path alone reaches 1.22 to 2.01 times on B200 and 1.17 to 2.00 times on B300 against the fastest baseline the authors evaluated. Code is published under the yandex-research GitHub org.

The authors are blunt about what breaks when you push attention into MXFP8 and mixed NVFP4/FP8: "causal information leakage through shared scales, and numerical inconsistencies between the forward and backward passes." A tiling mismatch of just 256 queries on the diagonal versus 128 pushed gradient norm from 0.5 to 2×10⁵ in one of their stress tests. Training loss read through a leaky kernel also looks fine until you re-evaluate the model under exact causal attention, where their reported numbers drift from 0.811 up to 0.940.

Validation ran on a 1B Llama-style model over 100B FineWeb-Edu tokens. Final loss: 2.8876 for the BF16 FA4 baseline, 2.8884 for MXFP8, 2.8885 for mixed NVFP4/FP8. The paper discloses that Claude and Codex assisted in writing the kernels, with the authors noting they "designed the methods, specified and reviewed every code change, and validated the code through tests and measurements." It lands in a busy stretch of open-source training-system releases on our tracker, among them EPFL and NYU's Library of Layers.