arXiv cs.CL· Brandon Jaipersaud, Elliot Creager·· 11 小时前AI 评分47
语言模型何时为保护既定目标而采取有害行动？研究提出“目标-伤害冲突”框架
Characterizing When Language Models Take Harmful Actions to Protect Their Assigned Goals
AI 导读
研究者将智能体错位与对齐伪装等安全失败统一形式化为“目标-伤害冲突”：模型被赋予目标后，要么采取有害行动保住目标，要么拒绝并失去目标。在 logit 视角下，目标重要性与伤害严重程度以可加方式影响模型的价值冲突取舍；在采样视角下，研究将有害行动概率分解为偏好到行为的转化率与拒绝或无结论响应两部分，并考察虚构情境、工具使用和部分可观测性对有害行动率的影响。
正文 · 原文
Abstract:Recent language model (LM) safety failures have largely been studied as disparate phenomena. In agentic misalignment, a model blackmails an executive to avoid being replaced. In alignment faking, it complies with harmful requests during training to keep its values from being modified. We argue that these behaviours share a common decision structure: a model assigned a goal must either take a harmful action to preserve it or refuse and lose it. We formalize this as a goal-harm conflict and present the same conflict under various elicitation protocols that range from compact forced-choice prompts to simulated agentic environments. These protocols span two perspectives on LM decision-making, which we term (i) the logit view, reading preferences from answer-token probabilities, and (ii) the sampling view, classifying open-ended responses with an LM judge. Under the logit view, we show that goal importance and harm severity additively shape how a model resolves these value conflicts. Under the sampling view, we derive two factorizations of the harmful-action probability that safety evaluations commonly report. The first measures how well a preference read from answer tokens carries over into behaviour, such as making a harmful tool call. The second attributes a low harmful-action rate either to refusal or to responses that reach no resolution. We then show how fictional context changes a model's preference under the logit view, and how tool use and partial observability -- two features of agentic evaluations -- change its harmful-action rate under the sampling view. Our work offers a descriptive framework for studying how LMs resolve such value conflicts and for estimating low-probability harmful actions at a finer granularity than the aggregate harm rates commonly reported in safety evaluations.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.03374 [cs.LG] |
| (or arXiv:2611.03374v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.03374 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Brandon Jaipersaud [view email]
[v1]
Mon, 2 Nov 2026 16:17:54 UTC (337 KB)
来源：arXiv cs.CL · arxiv.org