Cerebras Study: Layer Dropout Cuts LLM Training FLOPs 25%

Reintroducing layer dropout into large language model training can cut training FLOPs by up to 25% at matched validation loss, according to a new arxiv preprint from a Cerebras team led by Mostafa Elhoushi.

The paper reports more than 2,400 training experiments spanning models from 271M to 8.2B parameters and datasets up to 160B tokens, all on Cerebras CS-3 systems. It also claims 'up to 1.5x inference speedup with negligible accuracy loss' from post-training techniques the dropout enables: early exit, intermediate-layer skipping, and self-speculative decoding.

Layer dropout, also known as stochastic depth, is not new. What the authors argue is that it was dropped from modern pretraining recipes without much scrutiny. 'As models and datasets have scaled, dropout - particularly layer dropout - has largely disappeared from large language models (LLMs) pre-training recipes,' they write, adding that 'no comprehensive study has quantified, let alone mitigated, this effect.'

The headline efficiency claim is stated flatly: 'LLMs can achieve lower or similar validation loss while saving upto 25% of training FLOPs.'

Every pre-training run was on Cerebras CS-3 hardware, and the abstract publishes no per-benchmark numbers behind the phrase 'negligible accuracy loss.'