SynthDocBench Exposes Mid-Document Failures in Frontier VLMs

A new synthetic benchmark accepted at COLM 2026 puts a number on something document AI teams have been quietly suspecting. In SynthDocBench, researchers introduce a fully synthetic benchmark for long-context visual document understanding that independently varies document length, layout structure, modality composition, and question type, generated across six layout archetypes with a 40 percent random override to stop models from exploiting spurious correlations. Seven frontier vision language models were run through it.

The most concrete finding is positional. For five of six models, the middle third of a document is the hardest section, and five of six also show a negative Early-to-Late trend, with the steepest decline reported at 8.3 percentage points. The paper also calls out a breakdown of chart comprehension in long-document settings as a distinct failure mode that existing benchmarks cannot surface, alongside a sharp degradation as document length grows.

Why this matters if you are running enterprise document workflows: standard benchmarks like DocVQA, ChartQA, and MMLongBench-Doc combine length, layout, modality, and difficulty in ways that make failure hard to attribute, and the authors argue current models may be overfitting to benchmark artifacts rather than achieving robust long-context visual document understanding. If your retrieval pipeline routinely lands the answer in the middle of a long PDF, or the question depends on a chart deep in the document, this is where accuracy is quietly leaking, and no top-line score will tell you.

The honest caveat is that this is a fully synthetic benchmark, which is exactly what lets it isolate variables cleanly but also means real enterprise PDFs may behave differently in ways not captured here. The abstract does not name which seven VLMs were tested, does not quantify the chart-comprehension collapse, and does not identify the one model that avoided the middle-third failure. The forward-looking part is that a controlled diagnostic like this is what vendors need to close the gap, and buyers can now push for stress-test results on the specific dimensions that matter for their document workloads, not just leaderboard aggregates.