reddit.com via Reddit

llama.cpp PDL boosts throughput on Hopper and Blackwell

nvidia inference chips open source llama.cpp blackwell inference-optimization

Key insights

  • llama.cpp PR #22522 adds PDL support enabling dependent kernels to schedule before primary kernels finish on compute capability 90+ GPUs.
  • Throughput gains are confirmed exclusively on NVIDIA Hopper and Blackwell architectures; Ada Lovelace hardware sees no improvement.
  • The optimization targets large-batch inference workloads specifically, with SM utilization improvements as the primary mechanism.

Why this matters

Practitioners running high-throughput local inference on H100 or B100-class hardware now have a validated, low-friction path to throughput improvement by updating llama.cpp, with no model changes required. The architecture-specific nature of the gain sharpens the hardware procurement calculus for AI teams choosing between Hopper, Blackwell, and Ada-generation GPUs for inference clusters. Community-validated kernel scheduling optimizations in open-source inference runtimes closing the gap with proprietary serving stacks like TensorRT-LLM is a meaningful shift in the cost-performance equation for self-hosted inference.

Summary

llama.cpp's new Programmatic Dependent Launch support, merged via PR #22522, is delivering real throughput gains on NVIDIA Hopper and Blackwell GPUs, with community benchmarks now backing the claims. PDL lets dependent secondary GPU kernels begin scheduling before the primary kernel finishes executing. On architectures with compute capability 90 or higher, this reduces kernel launch overhead and squeezes more utilization out of streaming multiprocessors during large-batch inference runs. The gains are hardware-specific: Ada Lovelace GPUs, despite being recent, lack the scheduling pipeline required and see no benefit. Essentially: (NVIDIA, llama.cpp community) the gains are real but narrow. - PR #22522 is the specific merge enabling PDL; throughput improvements are confirmed on Hopper and Blackwell only - Large-batch inference workloads show the clearest uplift; single-request or small-batch workloads are unlikely to benefit proportionally - Ada Lovelace (compute capability 89) is explicitly excluded despite being newer consumer hardware than Hopper in some configurations For the open-source inference ecosystem, this marks one of the first community-validated, architecture-specific kernel scheduling optimizations landing in llama.cpp, suggesting Blackwell/Hopper owners running local inference at scale have a concrete reason to update.

Potential risks and opportunities

Risks

  • Users on Ada Lovelace hardware (RTX 4000-series, L40) who update expecting gains may misconfigure batch settings chasing a speedup that cannot materialize on their architecture.
  • If PDL scheduling exposes edge-case kernel synchronization bugs under specific model architectures or quantization schemes, production deployments updating without regression testing could hit silent accuracy or stability issues.
  • Narrow hardware targeting means the optimization widens the inference performance gap between organizations with Hopper/Blackwell clusters and those on older or consumer-grade hardware, concentrating throughput advantages further among well-capitalized operators.

Opportunities

  • Cloud providers and colocation operators with Hopper/Blackwell inventory (CoreWeave, Lambda Labs) can market PDL-enabled llama.cpp deployments as a differentiated offering for high-throughput open-source inference customers.
  • Quantization and inference optimization tooling vendors (Neural Magic, Unsloth) could integrate PDL-aware batch scheduling into their stacks to capture additional gains on top of existing weight compression techniques.
  • NVIDIA has a concrete incentive to publish detailed PDL tuning guides for llama.cpp workloads, reinforcing Blackwell's positioning against AMD MI300X in the open-source inference market where community benchmarks increasingly drive hardware decisions.

What we don't know yet

  • Magnitude of throughput gains across different model sizes and batch configurations is not yet systematically documented in the community benchmarks.
  • Whether PDL support will be backported or adapted for Ada Lovelace through alternative scheduling mechanisms remains unaddressed by the PR author.
  • No benchmarks yet compare PDL-enabled llama.cpp against TensorRT-LLM or vLLM on identical Blackwell hardware to quantify the remaining performance gap.

Originally reported by reddit.com

Read the original article →

Original headline: r/LocalLLaMA: Blackwell PDL Support in llama.cpp Delivers Measurable Throughput Gains — First Community Benchmarks on Hopper+ Hardware