Fraunhofer PFLM Predicts Six Languages From Synthetic Pretraining

A 300-million-parameter byte-level transformer trained with no real text now reads Wikipedia. In a preprint posted to Hugging Face papers, Lennart Carstens-Behrens and Holger Fröhlich of the Fraunhofer Institute for Algorithms and Scientific Computing SCAI describe the Prior-Fitted Language Model, or PFLM. Every pretraining sequence is drawn from a synthetic non-linguistic prior; the model never sees a word of any human language. Then its weights are frozen and it is fed Wikipedia bytes.

The headline number, as the abstract states: "On Wikipedia in six languages, bits per byte fall from the uniform eight to between 0.9 and 2.4 at one million bytes of context." The six are English, Chinese, Hindi, Arabic, Japanese, and Korean.

The pretraining task is meta-learning by construction. Each sequence comes from an independently sampled recurrent structural causal model, which the paper calls a synthetic language. "The model never sees the same language twice during training, so the only way to predict the continuation is to infer the language from the prefix," the authors write. The vocabulary is the 256 raw byte values; training ran for 150 billion tokens across six stages of increasing context length from 2^10 to 2^20 tokens. The backbone is a Qwen3-Next-style hybrid, with gated linear-attention layers and sliding-window attention in a 3:1 ratio.

The same frozen model does more than natural language. Given numerals in context it "learns to count, to compare magnitudes, and to add approximately." It predicts the Rudin-Shapiro sequence to 0.04 bits per symbol by 10^6 symbols, Kolakoski to 0.13, the prime indicator to 0.28. It compresses six non-text domains, from source code to speech, below gzip and PPMd.

"To the best of our knowledge, no prior work has shown a language model learning to predict natural language in context after pretraining only on samples from a constructed non-linguistic prior," the authors write. The framing is deliberate: "The model has not learned a language. It has learned to learn one."

The abstract does not pit PFLM against a 300M model conventionally pretrained on real text at matched context, so the result stands as a capability demonstration rather than a parity claim. It lands amid a run of prior-engineering papers moving through our open-source feed, including the EVA paper swapping VAE priors one day earlier.