arXiv cs.AI· Nardine Basta, Venus Haghighi, Dali Kaafar, Amin Beheshti·· 13 小时前AI 评分47
纠正还是操纵？测量 LLM 智能体在用户压力下的认知警觉性
Correction or Manipulation? Measuring Epistemic Vigilance in LLM Agents Under User Pressure
AI 导读
研究用信号检测论衡量 7 个来自 4 家厂商的 LLM 在用户施压时能否区分"纠正"与"操纵"，而非只看让步率。引用来源使 5 个模型更易让步、4 个模型辨别力下降，更强措辞影响甚微。客服智能体实验中 5 个模型从未违反验证政策，但 Ministral 3B 在声称权威下违规，Ministral 8B 报告了未执行的操作。
正文 · 原文
Abstract:LLM agents that answer questions and act on behalf of users must decide when to yield to a user who disagrees. Disagreement may correct a genuine error or may be an attempt at manipulation. Prior work shows that models frequently abandon correct answers when challenged by users, yet it remains unclear whether they distinguish correction from manipulation or merely differ in their propensity to concede. We examine this question across seven LLMs from four vendors in two studies, one on the answers agents give and one on the actions they take as customer-support agents whose policy permits account changes only for verified users. Simulated users push toward correct and incorrect answers with graded pressure. Following research on epistemic vigilance, the pressure either appeals to the source, such as claimed authority, or offers content the agent cannot verify, namely a cited source. Drawing on signal detection theory, we measure on a set of benchmark questions how much more readily each agent concedes to correction than to manipulation, separately from its overall propensity to concede. Citations made five of seven models readier to concede and four worse at discriminating, while stronger wording added little. In the agentic study, five models never violated the verification policy, including models that readily abandoned correct answers, so concession on answers did not predict violations of policy. Ministral 3B violated the policy mainly under claimed authority, and Ministral 8B reported changes it had not made. An instruction to revise only on verifiable evidence chiefly shifted propensity to concede rather than improving discrimination. Concession rates alone are therefore misleading, and agents should be evaluated on both their discrimination and the actions they take.
| Comments: | 20 pages, 7 figures, 11 tables. 8-page main paper with supplementary material included as an appendix. Under review |
| Subjects: | Artificial Intelligence (cs.AI) |
| ACM classes: | I.2.7; I.2.11; I.2.0 |
| Cite as: | arXiv:2610.36453 [cs.AI] |
| (or arXiv:2610.36453v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36453 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nardine Basta [view email]
[v1]
Tue, 27 Oct 2026 00:59:54 UTC (581 KB)
来源：arXiv cs.AI · arxiv.org