arXiv cs.AI· Lipeng He, Yihan Wang, Jiawen Zhang, N. Asokan·· 10 小时前AI 评分34
PIPETTE:面向鲁棒 LLM 智能体的提示注入紫队测试框架
PIPETTE: Prompt Injection Purple-Teaming for Robust LLM Agents
AI 导读
PIPETTE 是一个统一的提示注入紫队测试框架,能将红队测试发现的攻击转化为防御方的鲁棒性提升。它采用基于字典学习的奖励机制,鼓励红队测试覆盖更广的攻击策略,并集成面向间接提示注入防御的 on-policy 对抗训练流程。在 AgentDojo 和 ASB 基准上,PIPETTE 训练的防御模型对静态攻击的攻击成功率接近零,并能抵御强自适应攻击,同时良性任务效用接近未防御基线。
正文 · 原文
Abstract:Large language model (LLM) agents routinely interact with untrusted external data, exposing them to indirect prompt injection (IPI) attacks that hijack agents into executing attacker-chosen tasks. To uncover these vulnerabilities at scale, automated red-teaming methods often leverage reinforcement learning (RL) to generate attacks that achieve high attack success rates (ASR). Model developers use the attacks discovered during internal red-teaming to harden agents through adversarial training before deployment, a collaborative offensive-defensive process known in cybersecurity as purple-teaming. However, we show that achieving high ASR during red-teaming does not guarantee robustness gains from subsequent adversarial training. Specifically, we find that two state-of-the-art RL-based red-teaming methods, RL-Hammer and PISmith, suffer from mode collapse, repeatedly discovering similar attack strategies whose additional successes add no additional robustness improvements to adversarial training. To address this gap, we introduce PIPETTE, a unified prompt injection purple-teaming framework that effectively converts red-teaming discoveries into defender robustness improvements. PIPETTE employs a dictionary learning-based reward that encourages broader coverage of attack strategies during red-teaming. It also integrates an on-policy adversarial training pipeline tailored to IPI defense, enabling agents to resist attacks while completing legitimate user tasks. Extensive evaluations across agentic benchmarks (AgentDojo and ASB) demonstrate that PIPETTE produces defenders that achieve near-zero attack success rates on static attacks and can withstand strong adaptive attacks, while maintaining benign task utility close to that of the undefended baseline.
| Comments: | 18 pages, 6 figures |
| Subjects: | Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.24362 [cs.CR] |
| (or arXiv:2611.24362v1 [cs.CR] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24362 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lipeng He [view email]
[v1]
Wed, 18 Nov 2026 12:56:06 UTC (659 KB)
来源:arXiv cs.AI · arxiv.org