HKUST AdaExploreBench Caps LLM Agents Below 70% on Rule Drift

HKUST's AdaExploreBench, a benchmark that drops LLM agents into decision environments whose hidden rules change without warning, caps every one of eight leading models below 70% calibrated accuracy. Gemini-3.8-Flash leads at 67.25%; GPT-6 Sol trails at 61.87%; the open-source Llama-3.1-8B lands at 8.13%.

The authors from HKUST, UIUC, CUHK and CMU argue the bottleneck is not reasoning. Frontier agents 'can often reason effectively from informative evidence once it is available, but struggle to actively acquire the right evidence and recognize when previously acquired evidence is no longer valid,' the paper states. Supplying sufficient evidence lifts average performance by close to 10%. Forcing more probing moves it by -2.0%, with mean unique probes rising from 33.9 to 53.8 for no gain.

The adaptation failures are starker. In 36.3% of eligible episodes, agents report identical rule assessments before and after the latent rule changes. Factor-replacement drifts are recognized at 33.5%; direction-reversal drifts, where an existing factor's effect flips, land at just 10.8%, with the gap significant at p<10^-70. Average calibrated accuracy drops 26.9 points at the moment of rule drift and does not fully recover.

Change-recognition rate correlates with calibrated accuracy at r=0.928, close to the 0.916 correlation with mean unique probes. Probing harder is not the same as updating beliefs. The paper lands into a heavy run of agent benchmarks on our tracker this quarter.