Found first: a primary source the press has not covered yet.
Layer dropout was dropped from modern LLM training recipes without a systematic study of whether that was correct. "Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference", presented as a poster at ICML 2026, provides that study. With proper configuration, it achieves similar validation loss while saving up to 25% of training FLOPs, and enables post-training inference speedups of up to 1.5x with negligible accuracy loss.
What the source says
The study spans 2,400+ training runs on Cerebras CS-3 hardware, covering models from 271M to 8.2B parameters trained on datasets up to 160B tokens. Authors include Mostafa Elhoushi, Shane Bergsma, and Joel Hestness, among others. They systematically examined layer dropout distribution, scheduling, and optimizer hyperparameter tuning across scaling regimes. The key result: optimal layer dropout configuration yields lower validation loss at equivalent training compute, or equivalent loss at up to 25% fewer FLOPs. Dropout-trained models also support early exit, intermediate-layer skipping, and self-speculative decoding post-training, delivering up to 1.5x inference speedup.
Why it matters
Training FLOPs are the primary cost driver for frontier model development. A 25% reduction requires no new hardware, only changes to dropout configuration during training. The post-training inference gains are separate: models trained with layer dropout can be adapted for early exit and speculative decoding without retraining. Achieving the gains requires tuning dropout distribution, scheduling, and optimizer hyperparameters. The paper covers all three.