RExBench: top coding agent clears a third of research extensions

The best LLM coding agent tested on RExBench, a new benchmark of 12 realistic AI research-paper extensions, cleared about a third of the tasks autonomously, according to the ACL 2026 paper by Nicholas Edwards, Yukyung Lee, Yujun Audrey Mao, Yulu Qin, Sebastian Schuster and Najoung Kim. With human-written hints added, the top score climbed, but still stayed below 44%.

The authors evaluated 12 LLM agents across two frameworks, aider and OpenHands, against expert-authored extension instructions with automated evaluation. "All agents fail to autonomously implement the majority of the extensions," the paper reports, concluding that "current agents are still short of being able to handle realistic research extension tasks without substantial human guidance."

Three researchers we track posted the link the day it circulated.