paper web signal

IDEA flags backdoored GUI agents at inference, no trigger needed

TL;DR

  • IDEA detects backdoor activation in GUI agents at inference time without needing any trigger knowledge or attack data for calibration.
  • The method combines Intent Counterfactual Probing and Region Causal Audit and flags what the authors call 'intent-evidence dominance inversion.'
  • Experiments cover one model, Qwen2.5-VL-3B, across five attack families; the abstract claims a finite-sample false-positive guarantee and modest inference overhead.

Open-weight GUI agents downloaded from public hubs can pass clean-data acceptance tests and still misbehave the moment a visual trigger appears on screen. That is the supply-chain profile a group led by Zhizhuo Kou lay out in a paper on arXiv, proposing a runtime detector they call IDEA.

The pattern they look for has a specific name. "On clean steps, the instruction drives the action; when an unreferenced screen region hijacks the step, the instruction loses control over the decision state or that region dictates it instead," the authors write. They label this signature intent-evidence dominance inversion.

IDEA bundles two probes. Intent Counterfactual Probing measures how far redirected instructions move the decision state. Region Causal Audit measures how far ablating a candidate region moves it. A gated conformal score compares both displacements against clean steps, which the paper says provides "a finite-sample false-positive guarantee."

The evaluation is narrow. One model, Qwen2.5-VL-3B, across "five attack families." The authors report that IDEA outperforms state-of-the-art detectors, localizes the trigger region for every alarm, and adds "only modest inference overhead." The abstract publishes no per-attack-family accuracy or false-positive numbers.