arXiv paper: visual pretraining beats text-only baselines

A new arXiv paper from Yiming Zhang and colleagues takes a swing at one of the quietest assumptions in modern pretraining, which is that the right first step with a visually rich document is to throw the pixels away and keep the text. The authors argue the opposite. Their claim is that language models pretrained directly on document images consistently beat models pretrained on the text extracted from those same documents, across multiple backbones and multiple benchmarks.

The motivation is intuitive once you say it out loud. Figures, typeset equations, and page layouts carry information that text extraction flattens or loses, and current pipelines treat that loss as free. If it is not free, then a large fraction of frontier training data has been getting silently degraded on the way in. The paper frames this as a systematic study of unsupervised visual pretraining paradigms that skip the text-extraction step entirely.

The honest caveat is that the abstract, which is the load-bearing part of what is public right now, is qualitative. It says visual pretraining outperforms text-only pretraining, but it does not name the backbones, list the benchmarks, quantify the margin, or discuss compute cost per token. A preprint claim of consistent improvement across architectures is exactly the kind of result that either replicates cleanly and reshapes ingestion pipelines, or turns out to depend on a specific corpus, a specific backbone, or a specific evaluation. Treat the direction as interesting and the specifics as unsettled until independent runs land.

What is worth watching is who moves first. If a well-resourced open lab reproduces even a modest version of this on a public corpus with published numbers, the pressure on frontier labs to rerun their own pretraining ablations with visual inputs goes up quickly, and document-image tooling stops looking like an OCR side-quest and starts looking like part of the pretraining stack.