Bender-Gebru 'Stochastic Parrots' paper flags LLM scale costs

"How big is too big?" is the question Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell put to natural-language processing in "On the Dangers of Stochastic Parrots," published at the 2021 ACM FAccT conference and circulating on YouTube. The paper takes aim at what its abstract calls "BERT, its variants, GPT-2/3, and others, most recently Switch-C."

The title comes from a definition in section 6. A language model, the authors write, is "a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot."

The bill of costs is concrete. A Transformer trained with neural architecture search "generated 284t of CO2," measured against a "5t CO2e per year" human baseline. Benchmark gains are not cheap either: a 0.1 BLEU improvement on English-to-German translation "results in an increase of $150,000 compute cost."

The authors' recommendations start with "weighing the environmental and financial costs first," followed by curating and documenting training data rather than scraping the web, running pre-development stakeholder exercises, and directing research away from ever-larger language models. The full paper sits in the ACM Digital Library.