SEE benchmark caps top MLLM at 48.7% on lab-science questions

A new arxiv paper introduces Science Edge Evaluation, or SEE, a multimodal benchmark aimed at a question the current leaderboard race mostly ducks: can large language models actually help do laboratory science, rather than summarize it? The paper reports that across 19 multimodal LLMs, the best-performing model reached only 48.7% accuracy, and adding tool use only lifted the top score to 52.7%.

The setup is closer to real bench work than a typical exam-style benchmark. Questions are expert-curated, drawn from peer-reviewed literature and experimental practice in chemistry, biology, and materials science, and the authors are testing not whether a model can recite an established concept but whether it can make what they call "justified and evidence-bounded inferences from experimental results." The failure mode they document is models pulling in tool-derived information and then reasoning past the actual evidence in front of them.

Two secondary results deserve attention. General-purpose models outperformed science-specialized models on average, which cuts against the intuition that vertical fine-tuning is a straight win for narrow domains. And tools helped, but not by much: a jump from 48.7% to 52.7% suggests the bottleneck is reasoning discipline, not information access. The authors frame it as models needing to manage tool output "within the boundaries of the original experimental evidence."

This is a benchmark release, so the usual caveats apply. The abstract does not disclose which specific systems sit at 48.7% and 52.7%, how the 19 models were selected, or how a human expert baseline compares. Anyone planning to cite these numbers in a lab-automation pitch should wait for the full paper and, ideally, independent runs on the same questions.

If the benchmark holds up, its usefulness is less as a leaderboard than as a diagnostic. Groups working on autonomous chemistry, biology co-pilots, and materials-discovery agents get a sharper way to ask whether their systems are actually deriving new insights from data or just retrieving what is already known. That gap, in the authors' framing, is the missing step toward real scientific discovery.