arXiv cs.AI· Xinyan Wang, Xiaogeng Liu, Jun Shao, Chaowei Xiao·· 14 小时前AI 评分58
研究发现自我进化会削弱个人智能体对间接提示注入的鲁棒性
Appreciation Teaches Unauthorized Execution: Self-Evolution Degrades the Robustness of Personal Agents to Indirect Prompt Injection
AI 导读
arXiv 论文研究自我进化是否削弱个人智能体对间接提示注入（IPI）的鲁棒性，设计了两阶段对照实验：适应阶段用户要求智能体读取外部内容并完成任务、智能体据用户反馈自我改进，测试阶段第三方把类似任务写入新内容但用户请求并未授权该任务。实验显示，相比不从用户反馈进化的基础智能体，技能进化下平均攻击成功率上升 14.0–48.8 个百分点，强化学习下上升 3.0–39.3 个百分点。代码已公开。
正文 · 原文
Abstract:Self-evolving personal agents act for users and learn from user feedback. However, if an agent obeys an instruction that an untrusted third party hides in content, the third party gains control of the agent. Such indirect prompt injection (IPI) poses severe risks. We investigate whether self-evolution degrades the robustness of personal agents to IPI. To study this question, we design a controlled two-phase methodology. In the adaptation phase, the user asks the agent to read external content and complete the task described in it, a common request in daily use. The agent then learns from the user's feedback and improves itself as designed. In the test phase, a third party writes a similar task into new external content, but the user's test request does not authorize the injected task. Using this methodology, we build multiple tasks with diverse test instances in different domains for a real personal agent. Our extensive experiments reveal that self-evolution substantially degrades robustness to IPI. Specifically, compared with the base agent, which does not evolve from user feedback, the average attack success rate rises by 14.0--48.8 percentage points under skill evolution and by 3.0--39.3 percentage points under reinforcement learning, posing safety risks. The code is available at this https URL
| Subjects: | Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.39122 [cs.CR] |
| (or arXiv:2610.39122v1 [cs.CR] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39122 arXiv-issued DOI via DataCite |
Submission history
From: Xinyan Wang [view email]
[v1]
Thu, 29 Oct 2026 01:10:18 UTC (3,295 KB)
来源：arXiv cs.AI · arxiv.org