Nvidia NeMo-DCR Cuts 1T RL Weight Sync From 87.5 Min to 150 Sec

A full 1-trillion-parameter checkpoint transfer between AWS regions takes 87.5 minutes, according to a new NVIDIA paper posted on Hugging Face. The same refit, done with a system the authors call NeMo-DCR at a 3% weight-change rate, takes 150 seconds on a relay-tree transport.

The gap exists because almost nothing changes between RL steps. The paper's framing is blunt: "only 1% of BF16 weights change per step." Across Qwen3 and Nemotron models the authors measure 0.6–1.2% element change rates per step, with Qwen3 RL staying inside 1.5–2.0%.

NeMo-DCR encodes most of the changed weights as XOR masks against the version already resident on the rollout node, which the paper reports is 38–40% smaller than sending overwrites, and then compresses another 1.7–2.2× under zstd level-1. The receivers are promised identical parameter and buffer bits as a dense refit, which the paper labels "bit-exact."

The speedups scale with model size. 30B to 1T models land at 12–40× faster than a full-checkpoint transfer; 120B models hit 22.6 seconds on relay tree against a 750-second reference. Transport dominates refit latency at 77–94% of the total, so most of the remaining win would have to come from faster networks, not better encoding.

The reference implementation is roughly 7K lines of Python built on Megatron Bridge and the vLLM native loader, shipped as PR #2444 in the NeMo RL repo. It is NVIDIA's second agentic-RL infrastructure drop we've tracked this week.