MameLoshnLM: 8B Yiddish model built on Llama 3.1 lands at COLM

Yiddish rarely shows up in the modern LLM literature, so a paper landing at COLM 2026 that ships a dedicated 8-billion parameter Yiddish model, its pretraining corpus, and its evaluation benchmark all at once is worth pausing on. According to the arXiv paper from Uri Katz, Omer Goldman, Tomasz Limisiewicz, Reut Tsarfaty and Noah A. Smith, MameLoshnLM is a continued pretraining of Llama 3.1 8B, trained on a new corpus called Oytser that combines contemporary web sources with literary materials, and evaluated on a new multi-task suite called Kashes covering translation, linguistic analysis, information extraction and language understanding.

The gap the authors point at is one anyone who has tried to use general multilingual models on smaller languages will recognise. They describe the Yiddish that currently sits inside existing multilingual resources as "noisy, machine-translated, or misclassified" text. The implicit bet everywhere else in the field has been that pooling that kind of data with everything else and scaling up will eventually be enough. The claim here is the opposite: that a more deliberate corpus plus continued pretraining gives you a model that "better captures language-defining lexical and morphological patterns" and that MameLoshnLM "outperforms open baselines of similar scale."

The honest caveat is that those are the abstract's phrases, and the abstract is what we have this week. What the reporting does not give you yet is a public head-to-head against closed frontier systems on Kashes, the licensing story around Oytser, or how dialect variation across Litvish, Poylish and Hasidic Yiddish was handled during corpus construction. Take the win as reported for an 8B open model on its own benchmark, not as settled ground truth for Yiddish NLP.

What this release does give the field is a template. A curated corpus, a multi-task benchmark, and a continued-pretraining checkpoint, published together for a language with a deep literary record but almost no dedicated LLM infrastructure, is exactly the shape of release that other heritage-language communities can copy. If Oytser and Kashes hold up in use, the more interesting downstream story over the next year will be how many Ladino, Aramaic or Occitan groups follow the same recipe rather than waiting for a frontier lab to notice them.