Cohere megakernel serves 30B MoE 1.58× faster than vLLM

Cohere published a serving engine that runs its 30-billion-parameter North Mini Code model on a single NVIDIA H100 at 292 tokens per second at batch size 1, which the company measures as 1.58 times the throughput of vLLM on the same hardware.

The trick is that the whole decode loop is one CUDA kernel. Instead of launching a sequence of specialized kernels for QKV projection, attention, MoE routing, gather, GEMMs and normalization, Cohere fuses the sixteen operation types of a decode step into a single persistent kernel with a common threadblock ABI: 12 warps organized as 8 consumer, 1 controller, 1 producer, and 1 storer, with task descriptors of exactly 32 int32 fields. Synchronization runs through counters in global memory rather than kernel launch boundaries.

"Barriers are implemented as counters in global memory," the authors write. "A task waits on a single count, not on a specific upstream task, which makes both signalling and waiting for dependencies cost O(1) time, regardless of fan-in and fan-out."

The gains come from three sources the post ranks in order: eliminating idle streaming multiprocessors at kernel boundaries, letting operations start when their specific inputs are ready rather than when a wave completes, and prefetching immutable weights from HBM before dependencies satisfy. On the 132 SMs of an H100, the model streams 6.6 GB of weights plus roughly 0.5 GB of KV cache at 8K context. Cohere reports 62% of the 470 tok/s speed-of-light bound; vLLM sits at 39%.

End-to-end, on five reasoning and coding benchmarks, the megakernel ran AIME 2025 in 335 seconds against vLLM's 495, GPQA in 3,837 against 4,386, and SciCode in 8,712 against 11,006. Accuracy matched: 38.9% ± 1.6% on SciCode against 38.2% on vLLM, and 70.3% ± 1.1% on LiveCodeBench v6 on both engines.

The engine has a batch-size ceiling of 8, cannot run prefill and decode concurrently, and today targets one specific model on one specific GPU. The code is published as a single CUDA file at github.com/cohere-ai/cohere-megakernel. Cohere lists an RTX Blackwell port, FP8 and FP4 quantization, and tensor plus expert parallelism for multi-GPU as next.