Lab2Sim: 218 lab videos stump multimodal models on fidelity
TL;DR
- Lab2Sim bundles 218 chemistry, physics and biology experiment videos with 11,881 rubric items scoring fidelity and operability through browser execution.
- All eight tested systems clear a 93% Runnable Rate, but no direct baseline reaches 60% apparatus fidelity across the four rubric dimensions.
- EduVideo2Code lifts apparatus fidelity 24.15 points on GPT-5.6-Sol and 20.88 on GLM-5.3-Flash, though PCC gains vary sharply by backbone.
A benchmark released this week asks a plain question of today's multimodal models: can you watch a chemistry demo and turn it into a web page a student can actually operate? More than 93% of the pages the tested systems produced run in a browser. None of the six direct baselines reaches 60% on apparatus fidelity.
The benchmark, Lab2Sim, comes from South China University of Technology and The Hong Kong Polytechnic University. It bundles 218 chemistry, physics and biology experiment videos collected from Bilibili and YouTube, each paired with a video-grounded specification of its apparatus, ordered steps, preconditions and state transitions, instantiated into 11,881 rubric items. Human reviewers verify every draft specification against the source video, which the paper reports takes "about 13 minutes per video on average."
Six systems are tested directly: Seed-2.1-Pro, GLM-5.3-Flash, GPT-5.6-Sol, GPT-6-Astra, DeepSeek-V4.1-Flash and Kimi-K3. Apparatus Fidelity scores range from 20.62 to 61.32; Procedure Execution from 20.11 to 41.40. GPT-6-Astra leads on Procedural Constraint Compliance at 44.83%. The paper's framing is blunt: "A high Runnable Rate says little about fidelity."
The authors' own method, EduVideo2Code, runs a compile-verify-repair loop that extracts procedures from the video, compiles them into an explicit state machine, then executes the page in a browser and attempts repairs until a score threshold is met. On both tested backbones it lifts apparatus fidelity substantially, 20.88 points on GLM-5.3-Flash and 24.15 on GPT-5.6-Sol, and brings Runnable Rate to 100%. PCC gains are uneven: 22.00 points on GLM-5.3-Flash versus 1.19 on GPT-5.6-Sol.
Even the authors' method does not close the gap on subtle physics. In one case study, "condensation on the beaker wall is conveyed through text rather than a corresponding visual change." Dataset and code will be released at the project page. It joins a slim run of education-tagged alerts we have tracked over the past 90 days.
Originally reported by huggingface.co
Read the original article →Original headline: South China U and HK PolyU's Lab2Sim Turns 218 Experiment Videos Into Interactive Simulations