Study: VLMs need vision capacity, not a pretrained visual prior

The abstract of "Unbundling Multimodal Vision Encoding", by Ellis Brown and co-authors including Mike Lewis, Luke Zettlemoyer, Lucas Beyer and Saining Xie, argues the pretrained visual prior most vision-language models inherit from their upstream encoder is not actually load-bearing. The authors pretrain all models from scratch on common data and compare them at matched lifetime compute, isolating three design choices that a standard encoder bundles: "a visual prior from upstream pretraining, dedicated vision capacity, and pre-fusion placement of that capacity."

The headline result: "the visual prior is not necessary; an encoder trained jointly from scratch is on par with a pretrained one, without a separate training stage."

What does matter is capacity. "An encoder-free model trails on visual understanding even when its backbone is enlarged by the encoder's parameter count," the authors write. That capacity does not, however, have to sit in front of the language backbone. Vision-only modules added in-fusion to early backbone layers "perform comparably to a pre-fusion encoder and can lower time to first token."

The abstract publishes no per-benchmark numbers, no model scales, and no compute figures; the claims rest on the authors' controlled framework as reported.