reddit.com via Reddit

vLLM, SGLang, llama.cpp Benchmarked on Mixed Blackwell/Ada GPUs

inference open source inference ai-infrastructure benchmarks

Key insights

  • Throughput divergence across vLLM, SGLang, and llama.cpp is context-length and batch-size dependent on mixed Blackwell/Ada hardware.
  • Pipeline parallelism across architecturally mismatched GPUs produces engine-specific overhead not captured in homogeneous benchmark suites.
  • The RTX PRO 6000 96GB (Blackwell) paired with Ada cards represents a real deployment pattern lacking published performance baselines until now.

Why this matters

Practitioners building inference infrastructure today are inheriting Ada hardware while beginning to acquire Blackwell cards, making heterogeneous clusters the operational reality rather than an edge case, and these benchmarks are the first public data point for that exact scenario. Inference engine selection has direct cost and latency implications at scale, and choosing the wrong stack for a mixed-GPU fleet can mean 20-40% throughput loss without an obvious diagnostic signal. Founders and technical leads evaluating vLLM versus SGLang for production deployments now have community-sourced evidence that the answer depends on their specific hardware mix and workload shape, not just the engines' headline numbers.

Summary

Community benchmarks on heterogeneous GPU clusters are rare, and a LocalLLaMA developer just filled a real gap: hands-on throughput data from a 7-GPU setup mixing one RTX PRO 6000 96GB (Blackwell architecture) with Ada-generation cards, tested across vLLM, SGLang, and llama.cpp using pipeline parallelism. The results show meaningful throughput divergence across inference engines depending on context length and batch size, meaning the "best" engine on homogeneous hardware does not automatically win on mixed fleets. Pipeline parallelism across architecturally mismatched GPUs introduces coordination overhead that hits engines differently. Essentially: (vLLM, SGLang, llama.cpp) perform inconsistently on the same hybrid cluster depending on workload shape. - Long-context prefill is where the divergence is most pronounced, making engine choice load-sensitive rather than universally optimal. - The RTX PRO 6000 96GB is a Blackwell card with a large memory footprint, making it an attractive anchor in hybrid clusters that extend reach without full fleet upgrades. - No prior published benchmark covered this Blackwell/Ada mixed configuration, leaving practitioners to deploy blind. As Blackwell cards filter into labs and startups that already own Ada hardware, the question of which inference stack actually handles mixed-architecture pipelines is becoming a procurement and ops decision, not just a research curiosity.

Potential risks and opportunities

Risks

  • Teams that standardize on a single inference engine based on homogeneous-GPU benchmarks before validating on their actual mixed fleet could ship production systems with 30%+ throughput shortfalls that are difficult to attribute post-deployment.
  • vLLM and SGLang maintainers risk losing mindshare to llama.cpp among resource-constrained practitioners if community benchmarks consistently show the lighter-weight engine performing competitively on mixed hardware before the major frameworks optimize for it.
  • Hardware vendors (Nvidia) face indirect reputational pressure if Blackwell/Ada hybrid clusters show poor out-of-the-box software support, slowing enterprise adoption of incremental Blackwell upgrades into existing Ada deployments.

Opportunities

  • SGLang and vLLM contributors could capture practitioner loyalty by publishing official mixed-architecture tuning guides targeting Blackwell/Ada combinations, a gap the community is currently filling itself.
  • Cloud providers and GPU rental platforms (Lambda Labs, Vast.ai, CoreWeave) offering heterogeneous Blackwell/Ada node configurations could differentiate by pre-validating and publishing inference engine configs for their specific hardware mixes.
  • Inference optimization consultancies and MLOps tooling vendors (Baseten, Modal, Replicate) are positioned to offer heterogeneous-cluster profiling as a productized service as mixed-GPU deployments become the norm in 2026.

What we don't know yet

  • Whether SGLang and vLLM teams have acknowledged the mixed-architecture performance gap and have roadmap items targeting Blackwell/Ada pipeline parallelism as of May 2026.
  • Specific throughput numbers and latency percentiles (p50, p99) for each engine at each context length tested, which were summarized qualitatively in the post but not fully tabulated.
  • Whether the bottleneck on mixed-architecture clusters is PCIe interconnect bandwidth, pipeline bubble overhead, or memory transfer asymmetry between Blackwell and Ada cards.

Originally reported by reddit.com

Read the original article →

Original headline: r/LocalLLaMA: First Community Benchmarks of vLLM, SGLang, and llama.cpp on Heterogeneous 7-GPU Blackwell/Ada Cluster Using Pipeline Parallelism