NYU-Meta FAIR Paper Moves Vision Encoding Into the Backbone

A controlled study from NYU and Meta FAIR argues the standard vision encoder in a multimodal language model is really three design choices glued together, and two of them can be dropped without losing accuracy. In Unbundling Multimodal Vision Encoding, Chris Hoang, Mengye Ren and Saining Xie of NYU work with Meta FAIR's Hu Xu and collaborators including Mike Lewis, Marjan Ghazvininejad, Luke Zettlemoyer, Lucas Beyer and Jakob Verbeek to pretrain every variant from scratch at matched lifetime compute, across backbones from 0.6B to 6.4B parameters, on 16 Cambrian VQA benchmarks and 12 language benchmarks.

The first finding: the upstream pretraining stage is optional. "An encoder trained jointly from scratch is on par with a pretrained one, without a separate training stage," the abstract states. More encoder pretraining pushed ImageNet linear-probe accuracy from 80.4% to 84.4%, but VQA did not move. The paper is blunt that "encoder pretraining does not help even when its compute is free."

The second finding is where the encoder still earns its keep. "What matters instead is dedicated vision capacity," the authors write, and at 1.7B the encoder-free recipe trails the from-scratch encoder by 5.4 points on average VQA (44.4% versus 39.0%). Widening the backbone, deepening it, or doing both by the encoder's 0.4B parameter count does not close the gap; none of the tried mitigations gets within reach.

The third finding is the engineering payoff. "Vision-only modules added in-fusion to early backbone layers perform comparably to a pre-fusion encoder and can lower time to first token," the abstract reports, with vision-only FFN branches and Q/K/V/O projections placed in the first 8 of the 28 backbone layers. On an H200 with 16 concurrent requests, the in-fusion variant beats the pre-fusion encoder on time to first token at the same VQA. Encoder-free is fastest in that test but still trails on accuracy.

The authors flag their own limits. The encoders here were pretrained on the same images as the multimodal run, so "results do not rule out benefit from released encoders trained on additional data." Experiments stop at dense 6.4B backbones, with sparse mixture-of-experts flagged as the most compelling next step, and higher-resolution, text-rich training is deferred. Code and additional results sit on the project's companion site. It arrives in a busy week for multimodal work on our radar, alongside recent tracker entries on open vision models.