⚡ 168 h early
pretraining LMs on historical text paper
Pretraining Language Models on Historical Text
3 experts across 2 network communities independently surfaced this.
Why this matches
Research & technical analysis reaction
2 attributable expert contributions
· Ted Underwood, AI Firehose
“For fans of Talkie-1930 and all the methodological questions raised by historical models, here's a new entrant into the field, TypewriterLM, trained up to 1913. Corpus, instruction-tuning datasets, and event dataset are released. arxiv.org/abs/2606.02991” evidence ↗
3 experts
2 communities
1 sources clustered
“For fans of Talkie-1930 and all the methodological questions raised by historical models, here's a new entrant into the field, TypewriterLM, trained up to 1913. Corpus, instruction-tuning datasets, and event dataset are released. arxiv.org/abs/2606.02991”
“A study presents "TYPEWRITERLM," a 7.24B language model pre-trained on texts before 1913. It achieves strong language understanding without biases. The "HISTORY-EVENT" benchmark tests historical accuracy and model performance, aiding future research. https:…”
7 experts discussed this · 38 posts
Ted Underwood: For fans of Talkie-1930 and all the methodological questions raised by historical models, here's a new entrant into the field, TypewriterLM, trained up to 1913. Corpus, instruction-tuning datasets,…
Ted Underwood: Also the model itself. Inter alia, I am glad that I decided to work on a benchmark rather than a model.
Ted Underwood: Not as far as I know, and I haven't set it up locally yet, though that would be easy at this size.
Open the full discussion →