Zhejiang's RealMe lifts sparse captures to 2K volumetric video

Eight capture views, plus forty generated ones. That is the input budget RealMe, a SIGGRAPH Asia 2026 paper from Zhejiang University's State Key Laboratory of CAD&CG and 4DV.ai, uses to reconstruct a volumetric video at 2560x1440.

"This paper addresses the problem of creating immersive volumetric videos from sparse and low-resolution capture resources," the authors write. Conventional dense multi-view reconstruction, they argue, relies on costly capture setups that make it impractical in real-world scenarios. The pitch is to adapt a high-resolution image diffusion model into a 4D enhancer that cleans up what a cheap rig captures.

The base is FLUX.1 [dev], Black Forest Labs' 14-billion-parameter image diffusion model. On top of it the team adds layout-aware positional embeddings using 4-dimensional IDs for temporal and spatial layout, factorized spatio-temporal attention in place of full attention, and a latent frequency loss weighted at 0.1. A ControlNet branch and LoRA adapters are trained while the VAE encoder, text encoder, and remaining DiT parameters stay frozen. Training ran 96 hours on 8 NVIDIA H100 GPUs with bfloat16 mixed precision.

To feed it, the group curated RealVV: 100 sequences from 20 actors, 48 to 80 views per sequence, 300 frames per view, over 1.4 million multi-view images in total. They "plan to release it to the community." Baselines in the comparisons include FTGS, 4C4D, 4K4D, DIFIX3D+, Diffuman4D, STAR and FlashVSR.

The authors are upfront about the ceiling. "Due to the scarcity of high-quality multi-view datasets, our training data remains constrained, and the model shows limited generalization to large scenes or complex camera settings," reads the limitations section. The retrieved abstract and project page do not publish headline PSNR, SSIM or LPIPS tables against those baselines. It lands amid a dense run of recent AI video work pulling richer scenes from sparser inputs.