Audrey is leading our human-in-the-loop research agent evaluation project, as a follow-up to RExBench focusing on ideation capabilities. She's looking to recruit participants & get thoughts our experiment setup! aclanthology.org/2026.acl-lon...
- The best LLM coding agent on RExBench completed roughly 33% of 12 realistic research-paper extensions autonomously, with no hints.
- Adding human-written hints lifted the best success rate, but it still stayed below 44% across all agents tested.
- The benchmark evaluates 12 LLM agents across two frameworks, aider and OpenHands, on expert-authored extension tasks with automated scoring.