Korea University's SwitchMem Picks Text or Image Memory Per Step

The optimal memory modality for a GUI agent is step-dependent, not a design-time constant, according to a new paper from researchers at Korea University, Myongji University, and AIGEN Sciences posted to Hugging Face. Their system, SwitchMem, picks between a textual and visual rendering of the same retrieved past interaction at each step, and routes a frozen actor toward whichever representation would score higher.

The authors run the experiment on three mobile GUI benchmarks (Android in the Wild, LearnGUI, and MobileMiniWoB++) with Qwen3.5-9B held fixed as the actor across every memory-augmented method. "The preferred memory representation is strongly step dependent," the abstract states, and the step-wise oracle "consistently outperforms the strongest fixed-modality strategy across benchmarks."

Rather than train a new policy, SwitchMem fits a two-layer MLP router that reads the actor's hidden-state shift between the two memory conditions. The paper calls this a Directional Chain-of-Embedding signal; it becomes "most pronounced in the later transformer layers," peaking at layer 31 on AITW and layer 23 on LearnGUI. On MobileMiniWoB++, the LearnGUI-trained router is transferred without further training and lands at 63.043% task success (58/92).

The paired tests tell a mixed story. The improvement over the strongest fixed baseline is statistically significant on AITW (p<.001) and LearnGUI (p=.016), both surviving Holm correction. On MobileMiniWoB++ the gain is not significant (p=.688), which the authors attribute to the 92-task evaluation size and the lack of target-domain supervision. SwitchMem adds only 0.128s, 0.190s, and 0.304s per step over the Image baseline, versus 8.815s and 11.118s per step for the RAG-GUI baseline on AITW and LearnGUI.

The authors flag what the method cannot do. "Our online evaluation uses emulated environments rather than physical devices," they write, and routing requires actor hidden states, so it "cannot be directly applied to models accessed through APIs that do not expose these representations." Code and project resources are posted at the SwitchMem page.