paper web signal

WebArena-Pro: Coding Agents Double VLMs on Web Benchmark

TL;DR

  • WebArena-Pro spans 300 tasks across 20 self-hosted web apps modeled on services like Notion, Slack, Spotify, Figma, and Airbnb.
  • Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 top the benchmark at 62.3%; the strongest generic VLM agents reach 30.3%.
  • Even the leading configuration leaves 38% of tasks unsolved, and Gemma4 31B falls from 38.2% on single-site to 13.5% on cross-site tasks.

Coding-agent stacks roughly double the task-success rate of generic vision-language model agents on realistic browser work, according to the WebArena-Pro benchmark accepted at NeurIPS 2026. Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 both land at 62.3% on the 300-task evaluation. The strongest ReAct-style agents reach 30.3%.

The benchmark covers 20 self-hosted web applications across six domain categories (productivity, communication, analytics, business operations, design and media, and travel), each modeled on a widely used service such as Notion, Slack, Spotify, Figma, or Airbnb. It treats audio and video as core observations alongside text and vision, with dedicated actions for extracting information from each.

"Across the evaluated agents, the best performance is achieved by both Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 at 62.3%, followed by OpenCode with Kimi 3.1 Pro at 60.7%," the paper reports. The strongest GenericAgent configurations, Gemini 3.1 Pro omni and Gemma4 31B, "both achieve 30.3%."

The gap widens by task type. On communication tasks Codex reaches 81.8%; the Gemma4 31B generic agent "shows the largest gap, falling from 38.2% on single-site tasks to 13.5% on cross-site tasks."

The ceiling itself is unfinished work: "Even the strongest configuration leaves 38% of tasks without a recorded success."