Lucida rebuilds indoor scenes as editable assets from video

A new paper, Lucida, states its goal plainly. Take a video of a cluttered indoor room, and hand a robot simulator back an editable copy: each object a separate asset, placed where the camera actually saw it.

That order (parse the scene into instances, generate an asset for each, then place them) is the standard real-to-sim recipe. The authors argue every step of it presumes something a real capture rarely gives you: accurate instance geometry, unoccluded views, or assets that already match what was seen. Their pipeline keeps the order but 'redistributes the requirements, so each step consumes only what a real capture reliably provides and precision is reached at the end of the pipeline rather than demanded at its start.'

The unusual piece is at the end. Placement is done by 'GizmoAct, a VLM policy that casts placement as multi-turn GUI interaction, manipulating the object's gizmo in a closed loop and deciding itself when alignment is reached.' The vision-language model, in other words, drives the 3D editor's own transform handles and calls its own alignment done.

The reported gains: a 69% mAP improvement over Boxer on R2S-Scene for scene-level 3D object detection, [email protected] rising from 57.8% to 83.4% on CA-1M for object pose estimation, and scene F-Score climbing from 0.794 for SAM3D to 0.924. Per-instance and per-category breakdowns are absent from the abstract, and this is a preprint rather than a peer-reviewed result. Lucida arrives during a run of multimodal and computer vision research releases our tracker has been logging almost daily, 58 and 56 stories respectively in the last 90 days.