ScienceIDE Turns Scientific Code Repos Into Agent Environments

A team of 45 researchers has released ScienceIDE, infrastructure that turns scientific code repositories into executable environments where AI agents are trained and graded on whether their patches make physics simulations "numerically right again." The paper, submitted September 16, ships three model variants (PhAI-IDE-4B, PhAI-IDE-9B, and PhAI-IDE-72B) along with 15 of 64 environments and 30 of 85 tasks on GitHub.

The authors frame the target as a "scientific experience bottleneck": fragmented toolchains, implicit domain conventions and specialized correctness criteria that "make this knowledge difficult to convert into reliable learning experience," per the abstract. Instead of rewarding diff-matching, the verifier recompiles the codebase, runs a physics case, and compares numeric output against a reference.

Reported gains are largest under reinforcement learning. A Qwen3.5-4B baseline on the LAPS environment moves from 0.357 to 0.857 mean verifier reward, and on MITgcm-biogeo from 0.286 to 0.571. Supervised fine-tuning shows more modest lifts, with PhAI-IDE-9B climbing from 0.3125 to 0.500 on LAPS. Transfer to generic coding is uneven: the 9B posts +36.00 points on BBH Word Sorting, while the paper notes a regression on HumanEvalFix Python.

The release lands into a stretch of open post-training experiments we have tracked lately, from Xiaomi's public Mimo RL dashboard to Arcee's Series B on cheap open-weight training. ScienceIDE's held-out set of 49 environments and 55 tasks is not part of the initial drop.