Ulm Benchmark: LLMs Edit Vision Weights Better After ERT Prep
TL;DR
- Across more than 2,600 scored edit processes, Edit-Ready Training lifted Opus and Astra's net scores most on hard tasks like Entity-30 confusion fixes.
- Only frontier editors GPT-6 Astra and Claude Opus 5.5 reliably benefited from ERT; three open-weight editors showed no significant gain over standard training.
- Opus touched the classifier head in 296 of 308 processes, and across all training methods full-test accuracy ended 3-4 points below the unedited network.
A new benchmark from the University of Ulm reports that five LLMs ran more than 2,600 scored edits on image classifiers, and that a training recipe called Edit-Ready Training (ERT) lifts edit success for frontier models GPT-6 Astra and Claude Opus 5.5 where standard networks resist. The paper covers convolutional networks and vision transformers across Waterbirds, CIFAR-100, PACS and Entity-30.
ERT rehearses during training the closed-form head edits that LLMs tend to write at inference. The authors state that "ERT further achieves the highest mean score for GPT-6 Astra and Claude Opus 5.5 across all evaluated baselines." The largest gains sit on the hard settings: "On the Entity-30 confusion fix it raises the net score from 0.46 to 0.79 for Astra and from 0.44 to 0.74 for Opus," and on Waterbirds with ViT-B/16 from 0.50 to 0.74 and from 0.33 to 0.76.
Opus and Astra almost always touched the classifier head. The paper reports that "Astra changes its weights in 291 of 292 edit processes and Opus in 296 of 308." Three open-weight editors in the same protocol, GLM-5.3-Flash, Gemma-4-26B and DeepSeek-V4.1-Flash, did not show the same lift from ERT, with mean paired gains over standard training of -0.01, +0.06 and +0.03 and p-values above 0.3 in every case.
The gains also come with a cost. "For all training methods, accuracy on the full test sets still ends 3 to 4 points below the unedited network after the final edit," the authors write. They flag larger backbones, generative models and sequences of edits on one network as the natural next steps, and will release the testing framework alongside the 75,000 logged tool calls that describe scored edits from the five editors.
Originally reported by huggingface.co
Read the original article →Original headline: Ulm Benchmark Shows Edit-Ready Training Lets LLMs Reliably Rewrite Vision Model Weights