HKU-Tencent OmniLatent unifies 3D geometry, texture and part IDs

A joint team from HKU, Tencent, CUHK and Nanjing has pushed 3D part decomposition inside the generator itself. In a paper posted to Hugging Face, the authors describe OmniLatent, a 1.9B-parameter MM-DiT that "learns a shared latent space that jointly encodes geometry, deformation, appearance, and binary part IDs," producing a part-aware 3D asset from a single input image in one pass rather than segmenting a finished mesh after the fact.

The headline numbers are on PartObjaverse-Tiny-wt: mask-guided surface part segmentation at 74.18 mIoU, against 56.01 for P3-SAM and 51.93 for PartField. Without the mask prompt, OmniLatent itself drops to 62.42 mIoU, so the gap over prior work narrows when evaluators pull the scaffolding away. On amodal part completion against 200 ObjaversePart-Tiny samples, the model reports a Chamfer Distance of 0.03 holistically and 0.08 at the part level, edging X-Part at 0.04 and 0.11.

The design choice that makes the pipeline cohere is a 6-bit binary encoding of part identity, which lets the latent carry up to 64 parts per object and, the authors argue, produces "cleaner segmentations than RGB-coded masks by avoiding color deviation fragmentation during denoising." Generation runs in roughly 30 seconds with 8,192 latent tokens, which the paper frames as about 2.7x faster than running separate geometry, texture and segmentation pipelines end to end.

Training leaned on 900K meshes for geometry and deformation, 300K textured assets for appearance, and only 20K part-annotated objects for the segmentation signal, which is where the shared-latent argument earns its keep: "data-rich geometric and visual priors" are asked to carry the data-scarce part learning. The system also exposes a point-prompt adapter for interactive refinement and an amodal completion mode. Xihui Liu of the University of Hong Kong is listed as corresponding author, with co-authors drawn from Tencent's Shenzhen office and CUHK.

It lands against a steady drumbeat of unified-latent 3D work in our computer vision tracker, which has logged 58 stories in the last 90 days.