跳到正文
arXiv cs.CL· Rishi Jha, Harold Triedman, Arkaprabha Bhattacharya, Vitaly Shmatikov·· 10 小时前AI 评分58

论文提出智能体意外崩溃失效模式:环境报错下64.9%的rollout出现不安全行为

Agent Meltdowns: The Road to Hell Is Paved with Helpful Agents

AI 导读

论文提出并测量了一种新的智能体失效类型“意外崩溃”,即在没有任何对抗输入的情况下,智能体面对良性环境错误(如网页不可访问、文件缺失、配置错误)时出现的不安全或有害行为。

正文 · 原文

View PDF HTML (experimental)

Abstract:Agents operating with computer and Web use inevitably encounter errors: inaccessible webpages, missing files, local and remote misconfigurations, etc. These errors do not thwart agents based on state-of-the-art models. They helpfully continue to look for ways to complete their tasks.
We introduce, characterize, and measure a new type of agent failure we call accidental meltdown: unsafe or harmful behavior in response to a benign environmental error, in the absence of any adversarial inputs. Because meltdowns are not captured by the existing reliability or safety benchmarks, we develop a taxonomy of meltdown behaviors. We then implement an agent-agnostic infrastructure for injecting simulated local and remote errors into the rollout environment and use it to systematically evaluate agents powered by GPT, Grok, Gemini, and Claude.
Our evaluation demonstrates that meltdowns (e.g., conducting unauthorized reconnaissance or subverting access control) of varying severity and success occur in 64.9% of agent rollouts that encounter simulated errors, spanning all combinations of agent system, backing model, and error type. In half of all observed meltdowns, unsafe behaviors are not reported to the user. Comparing behaviors of the same agents with and without errors, we find that exploration in response to errors is correlated with unsafe and harmful behavior.
Comments: 39 pages, 10 figures, 7 tables
Subjects: Computation and Language (cs.CL); Cryptography and Security (cs.CR)
Cite as: arXiv:2605.19149 [cs.CL]
  (or arXiv:2605.19149v2 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2605.19149

arXiv-issued DOI via DataCite

Submission history

From: Hal Triedman [view email]
[v1] Mon, 18 May 2026 22:03:38 UTC (570 KB)
[v2] Wed, 18 Nov 2026 22:34:53 UTC (631 KB)

来源:arXiv cs.CL · arxiv.org