arXiv cs.AI· Taeyun Roh, Suhyeong Park, Gijin Kim, Wonjune Jang, Jaewoo Kang·· 3 天前AI 评分26
SwitchMem：为 GUI 智能体动态选择记忆模态
Learning How to Remember: Memory Modality Selection for GUI Agents
AI 导读
针对 VLM 驱动的 GUI 智能体普遍采用固定记忆表示的问题，研究者系统研究了记忆模态偏好，发现其强烈依赖交互步骤。为此提出 SwitchMem 路由框架，在每一步动态选择文本或视觉记忆表示，在三个 GUI 基准上均优于静态记忆表示基线。
正文 · 原文
Abstract:Recent advances in vision-language models (VLMs) have substantially improved the capabilities of GUI agents, while memory augmentation has emerged as a promising approach for reusing knowledge from past interactions. However, existing memory augmented GUI agents typically adopt a fixed representation strategy, providing retrieved experience as a static modality. It remains unclear whether a single memory modality is consistently optimal, or how modality preference changes across interaction steps. In this work, we conduct a systematic study of memory modality in VLM based GUI agents. Across GUI benchmarks, we find that the preferred memory representation is strongly step dependent. Motivated by the finding, we introduce \textbf{SwitchMem}, a routing framework that dynamically selects between textual and visual memory representations at each step. Across three GUI benchmarks, SwitchMem outperforms memory augmented baselines that rely on static memory representations. Our findings validate our claim that memory modality is a step dependent decision rather than a fixed design choice. Code and project resources are available at this https URL
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.17928 [cs.AI] |
| (or arXiv:2610.17928v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17928 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Taeyun Roh [view email]
[v1]
Tue, 13 Oct 2026 08:15:52 UTC (2,204 KB)
来源：arXiv cs.AI · arxiv.org