LittleLearner: a 5B LLM bounded to U.S. Grade-5 material

A new preprint trains a 5-billion-parameter language model on nothing but U.S. elementary school material, then asks what happens when you try to teach it anything above the fifth grade.

The paper, LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure, by Fanfei Li, Ryan Cotterell, Wieland Brendel and colleagues, introduces two artifacts. LITTLECURRICULUM is an 88-billion-token pretraining corpus that, in the authors' words, "explicitly [excludes] concepts, facts, and vocabulary taught above Grade 5." LITTLELEARNER is the 5B model trained from scratch on that corpus.

The point is not the model. The point is the sandbox. The authors describe it as "a developmentally restricted sandbox to study how models acquire, represent, and use data under a well-defined training scope." That is a rare condition in a field where nearly every base model has ingested a poorly characterized slice of the open web.

The first suite of experiments delivers a blunt result on the two most popular ways to bolt new knowledge onto an LLM. Post-training and in-context learning, the paper reports, "let LITTLELEARNER better utilize existing knowledge, but do not raise out-of-scope capabilities." Injecting facts is not the same as expanding what the model can do.

The abstract reports no per-benchmark numbers and does not describe how the Grade-5 cutoff was enforced across sources. Two researchers on our Who's Who list had already shared the paper on the day it went up.