arxiv.org web signal

CompAct-Bench: Even Frontier Agents Degrade Under Compaction

Agents Anthropic OpenAI ai-business

TL;DR

  • CompAct-Bench pauses successful agent runs, has the model under test compress history, then measures how much downstream task accuracy survives.
  • Claude- and GPT-family models perform strongest overall, yet substantial downstream accuracy loss persists under aggressive compaction, the authors report.
  • Failures include not just dropped information but distorted task state, with unresolved issues flipped into apparent conclusions or prior results forgotten.

Working-context compaction, the mid-task compression that long-horizon agents use to keep running past their input limits, remains "highly lossy even for frontier models", according to a new arXiv paper posted on November 19. The authors introduce CompAct-Bench, which pauses successful agent trajectories, has the model under evaluation compress everything that came before the pause, and lets the same executor continue the task to see how much ground is lost.

The benchmark pulls tasks from three places: information-seeking via BrowseComp, software engineering via SWE-bench, and a workspace-oriented internal suite the authors call CompanyBench. To keep comparisons fair across models, each compactor is evaluated under "controlled retention ratios using a lightweight iterative ratio-adjustment procedure". Claude- and GPT-family models come out strongest, but the paper is blunt that "substantial degradation in downstream task accuracy persists under aggressive compaction".

The sharper finding is about the shape of the errors. "Failures are often caused not only by omitted information, but also by distorted task state, such as turning unresolved issues into apparent conclusions or forgetting previously established results," the authors write, which "can lead to incorrect decisions or redundant execution". The abstract publishes no per-model accuracy figures, and CompanyBench is described only as internal. It arrives into a dense stretch of long-horizon agent evaluations hitting the arXiv.