paper web signal

Silent Bug in 52% of Open Gated-Model Checkpoints Silently Drops 20% of Training Steps

Summary

More than half of open-weight gated and hybrid model checkpoints carry a silent training defect: near-zero input embeddings that cause gradient amplification exceeding 30 orders of magnitude, causing PyTorch and HuggingFace to overflow their clipping routines and discard roughly one-fifth of all gradient updates with no error raised. Anyone fine-tuning Mamba, Gated DeltaNet, or xLSTM variants on new languages or domains may already be affected.