Nvidia JCA Co-Generates 3D Motion and Video for Any Rig

The paper's example is a monkey jumping a crossbar. A video generator can show the jump; what you cannot do today is "return that jump as replayable 3D motion placed in the 3D scene." Nvidia researchers argue their new model does.

Joints, Camera, Action!, by eight Nvidia authors (Noam Rotstein, Yoad Tewel, Chen Tessler, Haggai Maron, Gal Chechik, Or Litany, Yoni Kasten, and Yuval Atzmon), co-generates video alongside 3D motion for any supplied skeleton, plus a camera that registers that motion into the scene.

The design keeps the motion model separate. "We instead keep 3D motion in a separate model that supports arbitrary skeletons, and connect it to a pretrained video model through a small set of cross-attention layers," the authors write. The motion branch learns from motion-only data, which the paper notes is "easier to obtain and augment than paired video." A joint-localization head then predicts where each joint lives in the generated frames, and that is how the camera is recovered.

Training leans on a generative video editor to place animated 3D assets into "scenes suited to their actions," curated so "their known motion remains valid." Combined with human and robot video, that yields nearly 200K training pairs.

No benchmark numbers appear in the abstract. The paper lists capabilities (text-to-animation, video-to-motion, motion-to-video, and keyframe-based control) without quantitative comparisons, one of 45 video-generation items tracked on our feed in the last 90 days.