Found first: a primary source the press has not covered yet.
A paper submitted to arXiv on November 3, 2026 reports that more than half of untied gated and hybrid open-weight model checkpoints carry input embeddings with norms below 10−6, a defect that causes gradient clipping to silently overflow and discard training updates with no error raised. The paper, by Xin Liu, Boyuan Sun, Yuxin Wang, Haoran Song, Mingjian Zhao, and Mengyu Liu, surveys 291 open-weight language model repositories and proposes two targeted fixes.
What the source says
The authors surveyed 291 open-weight LLM repositories (280 distinct embedding tables) and found that 52.6% of untied gated and hybrid checkpoints (20 out of 38) contain reachable input embeddings with norms below 10−6. Zero weight-tied models were affected. Near-zero token embeddings amplify backward gradients by more than 30 orders of magnitude (up to 35), with individual gradient entries exceeding 1.84 × 1019. In Telugu continual pretraining of Nemotron-3-Nano-4B, the defect discarded updates on approximately 20% of training steps. Architectures confirmed affected include Mamba, Gated DeltaNet, and xLSTM.
Why it matters
Anyone fine-tuning a Mamba, Gated DeltaNet, or xLSTM variant on a new language or domain may be running with roughly one-fifth of gradient updates silently discarded, no warning raised by the training framework. The paper proposes two fixes: GroupClip, which uses 64-bit or rescaled norms for embedding gradient clipping, and GateFloor, which adds an offset δ ∈ [0.01, 0.1] to gate functions. Together they reduce problematic gradients by 28 orders of magnitude in experiments up to 370 million parameters, with perplexity comparable to the unpatched baseline. The 52.6% prevalence among untied gated checkpoints is high enough to warrant auditing embedding norm distributions before any continued pretraining run on these architectures.