arXiv cs.AI· Zirui Shang, Xin Shu, Yang Liu, Zhi Gao, Xinxiao Wu, Lifeng Fan·· 3 天前AI 评分23
面向应用流的持续 GUI 智能体学习的选择性知识控制
Selective Knowledge Control for Continual GUI Agent Learning over Application Streams
AI 导读
研究者提出激活条件选择性知识控制方法，通过神经元级梯度操作让 GUI 智能体在应用流上持续学习：对未激活神经元截断梯度以防干扰，对激活神经元做正交投影以兼顾稳定与适应，并在每阶段合并新识别的关键神经元。在多应用顺序基准上，该方法有效缓解了对旧应用的灾难性遗忘，同时保持对新应用的适应能力。
正文 · 原文
Abstract:Continual learning is a crucial capability for Graphical User Interface (GUI) agents to adapt to evolving applications while retaining knowledge acquired from previous applications. Such application streams pose a challenging knowledge modeling problem: new applications often share underlying knowledge with past ones, yet also introduce specific knowledge that must not interfere with historical knowledge. In this paper, we propose activation-conditioned selective knowledge control, a lightweight method that achieves selective knowledge retention via neuron-level gradient manipulation. Our method maintains a compact historical knowledge state to protect highly activated MLP neurons that preserve previous knowledge. When a new application arrives, it performs real-time gradient surgery conditioned on forward activation. Concretely, the protected neurons are categorized into two types: unactivated neurons holding specific knowledge, whose gradients are truncated to prevent interference; and activated neurons holding shared knowledge, whose gradients are orthogonally projected to preserve stability while enabling adaptation. After each application stage, newly identified critical neurons are merged into the historical state for future learning. Empirical evaluations on multi-app sequential benchmark demonstrate that our method effectively mitigates catastrophic forgetting on prior applications while sustaining robust adaptation to new ones.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.06530 [cs.CV] |
| (or arXiv:2609.06530v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2609.06530 arXiv-issued DOI via DataCite |
Submission history
From: Zirui Shang [view email]
[v1]
Sun, 6 Sep 2026 10:50:55 UTC (3,671 KB)
来源：arXiv cs.AI · arxiv.org