Augustinian BabyLM: visual seeds help words, not grammar

A small DeBERTa masked language model, trained on 10 million words, was given a head start. Before training began, visually grounded tokens received embeddings derived from the image regions they label. Every other token started random.

The imprint stuck. In a new arxiv preprint, Lisa Bylinina writes that "Visual initialization leaves a measurable imprint that lasts until the end of training." But on the standard BabyLM grammatical benchmarks, the effect is invisible. The only zero-shot win is object-property knowledge, measured by COMPS (Misra et al. 2023).

To probe further, Bylinina built a corpus-tailored version of the Visual-Property Swap benchmark from Lin et al. 2026, testing color, material, size and shape knowledge. Vision-seeded models had a "persistent, seed-replicated advantage, confined to the seeded words." A causal test showed that synthetically grounding previously unseeded words transferred the advantage to exactly those words.

There is one more finding the paper flags but cannot quite show. Function words and abstract vocabulary also receive strong visual seeds and keep them throughout training, and held-out mask-prediction loss falls for these words in every seed. No benchmark she runs registers it. "What evaluation would pick this up remains an open question."

The frame is Augustinian: St. Augustine's picture of word learning by ostension, encoded as an initial embedding rather than a training signal. Two researchers on our watchlist flagged the preprint within a day of posting.