Columbia, CertiK's TensorLean Verifies PTX, Finds 38 Kernel Bugs
TL;DR
- TensorLean verified 149 compiled GPU kernel instances, proved 83 correct and surfaced 38 previously unknown correctness bugs across six libraries.
- FlashAttention accounted for 23 of the bugs across 102 kernel instances from versions 2 to 4; DeepGEMM had two.
- The tool checks PTX assembly against a Lean tensor-level spec using Z3 and cvc5, with a spec-to-code ratio near 15:1, on A100, H100 and B200.
Researchers at Columbia University and CertiK ran a Lean-based verifier across 149 compiled GPU kernel instances and proved only 83 of them correct. In the other 66 they found 38 previously unknown correctness bugs, including 23 in FlashAttention and two in DeepGEMM, according to a write-up in MarkTechPost.
The tool is called TensorLean. Its authors, Wei Qiang, Ganxiang Yang, Yi Rong and Ronghui Gu at Columbia, with Xupeng Li at CertiK, describe it as "the first framework for formally verifying production GPU kernels." Instead of trusting source code, it checks the compiled PTX assembly against a tensor-level mathematical specification, proving that a kernel is "data-race free, terminates, and is functionally correct against the specification within floating-point tolerance."
One of the bugs is specific enough to recite. The team reports finding "a memory access error causing silent data corruption of KV cache in flash_attn_with_kvcache with non-interleaved rotary." That routine sits inside the decoding loop of many frontier LLMs.
The method pulls on more than one proof engine. "TensorLean writes these proofs in Lean, a proof language in which a small, trusted checker confirms every step," the project page says, with Z3 and cvc5 doing the SMT discharge. Spec-to-code sits at roughly a 15:1 ratio.
The six libraries covered include FlashAttention (102 instances across versions 2 to 4), DeepGEMM, Liger Kernel, Triton and TileOPs, running on PTX compiled for A100, H100 and B200. It is the sort of ground-truth claim that the AI infrastructure beat has had very little of this year.
Originally reported by marktechpost.com
Read the original article →Original headline: Columbia and CertiK Release TensorLean, Formally Verify 83 GPU Kernels at PTX Level and Expose 38 Bugs Including in FlashAttention