Video-Coder trains an agent to script 3D scenes for video

A new paper out of the University of Central Florida, the University of Würzburg, Microsoft Research, Tsinghua, UNC-Chapel Hill and UW-Madison reports that routing a video generator through an executable 3D draft improves MiniMax-H3 by approximately 7 points on PhyGenBench and 24 points on VBench-2.0.

Instead of asking a diffusion model to infer physics from pixels, Video-Coder trains an agent to write and execute Blender Python code that assembles a dynamic 3D scene. That scene is rendered as a 'Draft Video' which then conditions the downstream generator with explicit structure and motion signals. The authors describe the system, posted on Hugging Face, as 'the first attempt to train a code-driven video generation agent, which autonomously writes and executes code to construct a dynamic 3D scene.'

The data pipeline hangs on two new sets, Video-Coder-SFT-6k and Video-Coder-RL-2k, built by wiring Gemini 3.8 Flash into a custom Video-Coder Harness that runs code, renders 3D scenes and returns execution feedback inside one loop. Supervised fine-tuning teaches the base scene-programming moves; a second stage of agentic reinforcement learning then optimises two rewards at once, one for whether the generated program actually runs, one for the quality of the finished video.

An annealing schedule, the paper reports, 'gradually reduces the execution reward over time, allowing the model to rely more on the video reward in the later training stages.'

The abstract reports only averaged benchmark deltas on a single backbone, MiniMax-H3; per-prompt failure modes and comparisons against other generators are not quoted there. The team says it has 'fully open-source[d]' its data, models and code. It lands amid a run of agentic-pipeline releases tracked in our generative AI feed.