Paper: Gated LMs Silently Freeze 20% of Multilingual Steps

Across 291 open-weight language model repositories, 52.6% of untied gated and hybrid checkpoints (20 of 38, including Nemotron-3-Nano-4B, Falcon-H1, Mamba-Codestral-7B, xLSTM-7B and Kimi-Linear-48B-Base) carry input embedding rows with norms below 10^-6, according to a paper posted on Hugging Face. None of the 80 weight-tied Transformers in the survey showed the pattern.

Those dormant rows arise during pretraining when rare or template tokens never appear in training text, while AdamW weight decay keeps shrinking their untied embeddings. The paper reports that in gated sequence models (Mamba, Gated DeltaNet, xLSTM), hitting one of these rows amplifies the input-embedding gradient by 'over 30 (up to 35) orders of magnitude.' Once any entry crosses 1.84×10^19, FP32 global-norm clipping overflows to infinity, and 'PyTorch clip_grad_norm_, HuggingFace Trainer, and FSDP2 log grad_norm = inf without error and apply zero gradient updates.'

The downstream cost is concrete. In a 1,000-step Telugu continual pretraining run on Nemotron-3-Nano-4B, 22.5% of batches triggered overflow and 19.8% of steps lost their updates. Trigger rates sit at 0.0% on English web, English SFT, code and math, but reach 95.4% on the Telugu corpus, 100% on Falcon-H1 Mongolian, and 100% when Kimi-Linear-48B-Base sees the `<|im_user|>` chat tag at position zero.

The authors propose a three-line fix they call GroupClip, which norms the embedding gradient in FP64 and clips it separately from the backbone. On the Telugu run it drops stalled steps from 198.3 to zero, lifts the triggered Telugu rows from ≤2.3×10^-10 to a median of 0.010, and reaches a lower final perplexity (13.04 vs 13.44) than FP32 global clipping. Step time barely moves, from 2.88s to 2.84s. For from-scratch pretraining a 'GateFloor' offset of δ=0.1 on the SiLU gate cuts the r=10^-30 gradient of a Mamba-2 165M from 5.6×10^28 to 0.29, with matched perplexity.

LoRA fine-tuning and on-policy RL are unaffected, because both read from the output embedding. Megatron-Core's check_for_nan_in_grad raises; the other frameworks do not.