LabInstruct benchmark: video models fail lab procedure tests

Frontier image-to-video models can produce laboratory demonstrations that look right and still get the procedure wrong, according to LabInstruct, a new benchmark for situated instructional video generation in real labs. The authors evaluated eight frontier image-to-video models on 204 tasks across five scientific disciplines, each paired with real reference executions and structured annotations of objects, actions, contacts, and state transitions.

The finding is blunt. "Visually plausible generations frequently remain procedurally incorrect," the paper reports, "revealing a substantial gap between visual realism and the reliability required for experimental instruction."

The motivation is the self-driving laboratory, where AI-generated plans are meant to be carried out by robots or by human technicians following machine-generated demos. The authors argue humans remain "an important execution interface between AI-generated plans and physical experiments," which is why they treat procedural video as the thing to measure. The abstract names no specific models and publishes no per-model scores; what it puts forward is the 204-task benchmark and the gap it exposes.