TurboServe cuts streaming video latency 37.5% on B300 GPUs

Streaming video generation has quietly become its own serving-systems problem. Users hold a long-lived session while a model produces the video in chunks, each one owed to the user under a tight latency target, and that shape does not match either offline batch video generation or the request-and-forget rhythm of typical LLM serving. A new paper on arXiv, TurboServe: Serving Streaming Video Generation Efficiently and Economically, argues that this workload deserves its own scheduler and presents itself as the first serving system designed specifically for it.

The authors frame the problem around two headaches that anyone running a fleet will recognise. Sessions live for very different amounts of time, so a placement decision that looked good when a session arrived can quietly go stale, and the number of active sessions swings sharply between bursts and idle periods. TurboServe's answer is a closed-loop online scheduler that combines a migration-aware placement controller, which rebalances sessions across GPUs to hold down the maximum per-chunk latency, with a load-driven autoscaling controller that adjusts how many GPUs are on the clock. Underneath sit three enabling pieces the paper names explicitly: coalesced chunk processing for batching concurrent sessions on the same GPU, GPU-CPU offloading for suspending and resuming sessions, and NCCL-based GPU-GPU migration for online rebalancing.

The headline numbers, reported against unspecified 'baseline serving configurations', are a 37.5% reduction in worst-case per-chunk latency and a 37.2% average reduction in total GPU operating cost. Both come from an evaluation on real-world production traces from Shengshu Technology, across multiple model sizes and GPU clusters with up to 64 NVIDIA B300 GPUs, which is a heavier stack than most academic serving papers get to touch.

The honest caveat is that the abstract does not name the baselines those percentages are measured against, and the traces come from a single provider, so the size of the win on other traffic patterns is an open question. What the reporting also does not give you is the absolute latency floor per chunk, which is what actually determines whether the user experience feels live. Still, the direction is the interesting part: as streaming video products keep shipping, the teams renting B300 capacity and the researchers building interactive audio or agent-loop workloads now have a concrete template for scheduling long-lived generative sessions rather than treating them as awkward LLM traffic.