Talkie: a 13B language model trained only on pre-1931 text

Nick Levine, David Duvenaud, and Alec Radford have released talkie, a 13B-parameter language model trained exclusively on 260B tokens of pre-1931 English text, under an Apache 2.0 licence. The team ships a base model of 53.1 GB and an instruction-tuned chat variant of 26.6 GB.

The point of a frozen historical corpus is to ask questions that frontier models cannot answer cleanly. The project names three: "How good are these models at predicting the future?", "could a model trained up to 1911 independently discover General Relativity, as Einstein did in 1915?", and "Can they be taught to program?" For the first, they report measuring "the surprisingness of short descriptions of historical events to a 13B model trained on pre-1931 text".

The purity breaks down at post-training. The chat version was fine-tuned "using Claude Sonnet 4.6 as a judge" over "multi-turn synthetic chats between Claude Opus 4.6 and talkie". The authors concede the point: "While we have tried to post-train talkie free from modern influence, reinforcement learning with AI feedback inevitably shapes talkie's behavior anachronistically." Simon Willison, who posted the release on his weblog, prompted the model for an SVG of a pelican on a bicycle. It replied: "An SVG of a pelican riding a bicycle was generated in 1860."