SwitchMem · GUI Agent Memory
1Korea University · 2Myongji University · 3AIGEN Sciences
nrbsld@korea.ac.kr · kangj@korea.ac.kr
The same experience can help an agent differently when presented as text or an image. SwitchMem learns to select the memory modality that fits each interaction step, giving a frozen GUI agent the right representation of its past experience.
Problem
GUI agents reuse demonstrations to navigate unfamiliar interfaces and avoid repeated errors. Most memory systems commit to a fixed representation: text, screenshots, or both together.
Text offers a compact semantic abstraction, while images preserve layout and appearance. Their usefulness changes with the current interface and the action the agent needs to take. SwitchMem treats this choice as part of the interaction loop.
Motivating Observation
With the retrieved demonstration held fixed, Text and Image each win on substantial subsets of interaction steps. A step-wise oracle reveals the headroom left by a fixed memory policy.
Share of non-tied steps by preferred modality.
Text memory Image memory
Figure 2 in the paper.
Action score (%), best fixed modality → step-wise oracle.
Figure 3 in the paper. The oracle uses ground-truth scores to choose between two generated actions; it is an analysis reference.
Method
SwitchMem uses the actor's internal response to memory modality as supervision. A lightweight pairwise router then predicts this preference from the current GUI state and the retrieved memory candidates.
Run the frozen actor with Text and Image versions of the same retrieved demonstration. Keep the signed hidden-state shift hI − hT at the final prompt token. Deeper layers expose a stronger modality-preference signal.
Standardize the final-layer shift, reduce it to 64 dimensions with PCA, and fit a logistic probe on sampled action preferences. Its continuous decision value becomes the target for training a shared MLP scorer.
Score both memory candidates against the current-state representation. Select Image when the score difference reaches the calibrated threshold; otherwise select Text. Pass the chosen memory to the frozen actor to generate its next action.
r = qI − qTr ≥ τ: Image memory / otherwise: Text memory
The main experiments use a frozen Qwen3.5-9B actor and Qwen3-VL-Embedding-2B for retrieval and memory embeddings. Both candidates represent the same top-1 retrieved demonstration.
Main Results
SwitchMem improves over every fixed memory condition and the evaluated memory-augmentation baselines with the same frozen Qwen3.5-9B actor.
+2.120 points over Image
+0.690 points over Image
+2.173 points over Both
Table 1 in the paper. All rows below use Qwen3.5-9B. Scores are percentages; higher is better.
| Method | AITW | LearnGUI | MobileMiniWoB++ |
|---|---|---|---|
| No memory | 41.920 | 23.485 | 58.696 |
| RAG-GUI | 18.300 | 24.186 | 9.783 |
| M3A | 17.200 | 20.899 | 41.304 |
| AppAgent | 14.780 | 31.104 | 25.000 |
| Text memory | 48.760 | 35.638 | 53.261 |
| Image memory | 49.300 | 37.441 | 57.609 |
| Both modalities | 47.080 | 26.919 | 60.870 |
| EchoTrail | 45.900 | 27.286 | 55.435 |
| AGMem | 45.380 | 28.366 | 52.174 |
| SwitchMem (Ours) | 51.420 | 38.131 | 63.043 |
AITW and LearnGUI use benchmark-specific step-level action scores. MobileMiniWoB++ measures closed-loop task success (58/92 tasks). Its router weights are transferred from LearnGUI; the threshold is calibrated without actions, rewards, or task-success labels.
Routing Behavior
SwitchMem frequently uses both modalities over the course of a task, while retaining a chosen modality for consecutive steps. Its policy adapts to interface state rather than assigning one modality to the entire task.
Table 2 in the paper. Dual-modality trajectories use Text on some steps and Image on others.
| Benchmark | Dual-modality trajectories (%) | Image-routed steps (%) | Mean switches per trajectory | Mean run length (steps) |
|---|---|---|---|---|
| AITW | 83.8 | 36.5 | 1.72 | 2.42 |
| LearnGUI | 90.8 | 81.3 | 1.79 | 4.95 |
| MobileMiniWoB++ | 57.8 | 76.3 | 1.56 | 4.50 |
Analysis
Actor-derived preference supervision outperforms simpler routing targets. SwitchMem also beats ratio-matched random selection at every evaluated intermediate Image-use rate on LearnGUI: gains come from assigning modalities to appropriate steps.
Image memory becomes more useful when fine-grained layouts contain dense UI elements. SwitchMem increases Image selection in these states.
The Image-over-Text utility gap grows for actions governing interface functionality, task status, or system navigation.
Image memory becomes more useful after the initial state, as interface changes introduce visual details that textual abstraction may lose.
At inference, SwitchMem obtains a current-state hidden representation for routing and then generates an action with the selected memory. It avoids evaluating both memory-conditioned action prompts online. The latency comparison uses the same RTX PRO 6000 GPU for all methods.
Generalization
Applying the preference-construction and routing protocol to UI-TARS-7B-SFT also produces the best scores across all three benchmarks, exceeding its strongest fixed memory condition.
Table 3 in the paper. UI-TARS-7B-SFT scores (%).
| Memory | AITW | LearnGUI | MobileMiniWoB++ |
|---|---|---|---|
| No memory | 29.980 | 25.705 | 23.913 |
| Text | 32.640 | 38.230 | 32.609 |
| Image | 30.440 | 40.997 | 33.696 |
| SwitchMem (Ours) | 33.680 | 41.055 | 36.957 |
Takeaway
Retrieving a relevant experience is only part of the problem. Choosing its representation matters too. Memory modality preference varies across steps and is reflected in the actor's internal representations.
SwitchMem turns that signal into a practical routing policy, combining the complementary strengths of text and image memory while providing a single memory modality for each action.
Resources