HKBU's VisualPatchWorld Beats Code Baselines by 23.5 Points

A new paper from Hong Kong Baptist University makes a specific claim worth reading carefully: you can induce executable, editable Python code that models a robot's world dynamics well enough to plan with, and beat the strongest LLM-code baselines by a wide margin, provided you split the problem into two steps rather than asking a language model to freewrite the code end to end.

The method, called VisualPatchWorld, first selects a qualitative dynamical form via short active probes (is motion linear or nonlinear, is object motion gripper-gated, are there contact forces), and only then fits the free parameters under a multi-step rollout loss rather than single-step prediction. On four control domains, Two-room, Reacher, PushT and Cube, the reported mean planning success is 69.0%, versus 45.5% for POMDP-Coder and PatchWorld, a 23.5-point gap. On the arm-reaching task, success climbs from 8% to 72%.

The most striking single number is on contact-rich pushing. PushT is where pure induced code stalls at 22%, but a hybrid variant that lets the ground-truth engine re-score the top 30% of candidates lifts that to 88-96% while cutting physics queries by 70%. The practical pitch is getting most of the accuracy of a full simulator without paying for a simulator query on every candidate the planner considers.

The honest caveat is that this is a single-author preprint from one lab, benchmarked on four synthetic control tasks, and the authors themselves flag that induced code alone is not enough for contact-rich manipulation, which is exactly the regime where robot planning gets hard. The reporting also does not tell you how the hand-designed sketch library behaves when no template fits, or how the pipeline holds up on real hardware with noisy perception rather than oracle or tool-extracted scene graphs.

What is worth watching is the direction. If the useful part of a world model is a compact, inspectable program rather than a giant neural blob, a class of robot tasks becomes debuggable in ways neural world models are not, and the door opens for teams without large-scale training budgets to still get planner-usable dynamics from offline traces.