PointDiT paper: plain ViT diffusion tops complex 3D geometry models

A new ICML 2026 paper on arxiv is making a fairly blunt claim about monocular 3D geometry: the elaborate machinery the field has built around latent diffusion and hybrid losses is not actually pulling its weight. The authors introduce PointDiT, and say a plain Vision Transformer diffusion model, operating directly on raw 3D point map patches, does the job better.

The setup they describe is deliberately stripped down. It is a pixel-space Diffusion Transformer built on a plain ViT, conditioned on image tokens from a pre-trained DINOv3 encoder, trained entirely from scratch. No point map tokenizer, no compression into a latent space, no hybrid loss stack. In the abstract's own words, "such architectural overhead and intricate loss formulations are unnecessary." The authors listed include Haofei Xu, Rundi Wu, Philipp Henzler, Nikolai Kalischek, Michael Oechsle, Fabian Manhardt, Marc Pollefeys, Andreas Geiger, Federico Tombari and Michael Niemeyer, which is a heavyweight bench for a claim this deflationary.

The reason a result like this is worth paying attention to, even before you dig into the numbers, is that the current monocular-3D playbook mostly imported ideas from image diffusion: compress into a latent space, reuse a pre-trained latent diffusion backbone, patch geometry-specific losses on top. If a plain ViT trained from scratch on point maps can "surpass complex latent-based diffusion models while remaining significantly simpler than hybrid alternatives," as the paper claims, then a lot of engineering complexity that geometry teams inherited by analogy is optional. The abstract also flags sharper geometric structure and better robustness in "highly ambiguous regions, such as transparent objects," which is exactly where existing depth and point-map models tend to fall over.

The honest caveat is that the abstract is what I have to work with here. It does not name the datasets, does not give quantitative comparisons, and does not disclose inference cost or model size, so the "surpasses" claim is the authors' framing rather than a settled benchmark result. Take the specifics as reported, not as replicated.

What is worth watching is whether the rest of the geometry stack, and the 3D generative pipelines built on top of it, quietly starts dropping tokenizers and bespoke losses over the next few conference cycles. Simpler wins tend to compound.