Cohere open-sources decode megakernel, 1.58× vLLM on one H100

Cohere published a serving engine that runs the entire decode forward pass as one persistent CUDA kernel. It does 292 tokens per second on a single H100 at batch size 1 in BF16, which is 62% of the hardware's theoretical bandwidth ceiling.

The cohere-megakernel repo frames itself plainly: "A fully-fledged serving system built around a decode megakernel." The trick, per AlphaSignal's write-up, is "One threadblock per SM resident for the entire decode step," with counter-based barriers in global memory replacing per-operation kernel launches.

Against vLLM v0.24 on the same H100, the decode step runs 1.58× faster at batch size 1. End-to-end, the speedups shrink but hold: 1.41× on AIME 2025, 1.37× on SciCode, 1.33× on MMLU-Pro (CS), 1.28× on LiveCodeBench v6, 1.25× on GPQA. Accuracy stays flat. SciCode comes in at 38.9% for the megakernel against vLLM's 38.2%; both tie at 70.3% on LiveCodeBench v6.

The catches are stacked in the README. It runs on H100 SM_90a only, BF16 only, batch sizes 1 through 8, and is decode-only. Prefill still falls back to standard PyTorch kernels and pauses decode while it runs. It is also model-locked to North Mini Code, Cohere's 30B sparse mixture-of-experts with 3.3B active parameters, streaming 6.6 GB of weights per decode step. Apache 2.0.