Alibaba's FocusGUI boosts omni-modal GUI agents 33.2 TM points

A lightweight adapter called FocusGUI raises overall Type Match by 33.2 points and Exact Match by a factor of 3.0 on average across four frozen omni-modal GUI-agent backbones, according to an Alibaba paper posted to arXiv on 20 October. The adapter also beats the strongest existing enhancement method by 30.1 Type Match points, and trains from next-action supervision alone, with no modality-relevance labels and no fine-tuning of the underlying backbone.

The paper names the failure mode "Sequential Distraction": once an agent misreads which input matters at one step, its executed action feeds back into the context of the next step, so errors compound into trajectory drift. Figure 1 reports that "the dominant modality switches across 45.3% of adjacent steps, with 94.0% of episodes containing at least one switch", and that "90.1% of steps contain at least one interfering modality". The decisive target element occupies a median of just 0.87% of the screen area, so a decision that depends on audio, video, or device state cannot be recovered by cropping the screen.

FocusGUI exposes the observation as separable perceptual channels, scores each against a step-specific decision reference, compresses the result into a 64-dimensional latent, and injects it as a zero-initialized residual over the four prompt-embedding positions just before the action boundary. Between 1.15 million and 1.45 million trainable parameters sit outside each frozen backbone, trained for three epochs with AdamW at a learning rate of 2×10⁻⁴ and weight decay of 10⁻³.

The margin widens as trajectories lengthen. On Qwen2.5-Omni, the frozen backbone falls from 64.79 Type Match at the opening step to 32.79 beyond the third, while FocusGUI retains 86.8% of its initial accuracy and its Exact Match actually rises from 38.73 to 46.72. Screen-refinement baselines DRS-GUI, OmniParser, and RegionFocus come out near-neutral on Type Match but lower Exact Match by 28% and Goal Progress by 35% in relative terms, the paper reports, because none of them can reach the audio, video, or device-state channels. It arrives in the same week as Alibaba's OpenGCUA GUI-agent paper, another frozen-backbone adapter from the lab.