arXiv cs.CL· Junwei Yao, Jianan Liu, Yongxi Zhou, Zike Yang·· 11 小时前AI 评分42
研究：低置信度监控能排序提示词注入劫持，却拦不住它们
Ranked but Not Stopped: Detecting Prompt-Injection Hijacks from Typed Decision Models' Output Probabilities
AI 导读
针对把语言模型当分类器使用的场景，研究者用模型自身置信度检测提示词注入劫持，在 AG News 主题分类的五个系统（三个经 API 提供）上测试：低置信度在五个系统中有四个能把被劫持答案排在干净答案之前（AUROC 0.86-0.90），但排序不等于拦截。
正文 · 原文
Abstract:Programs that use a language model as a classifier act on its most probable label. Text injected into the input can move that label to one the attacker chooses, a hijack. A monitor that does not know the attacker's label can read the model's own confidence and raise an alarm when the top probability is low. We test this monitor on AG News topic classification with five systems, three of them served through a provider's API. Pooled over five families of injected text, low confidence ranks hijacked answers above clean ones on four of the five systems (AUROC 0.86-0.90). Ranking them does not stop many of them. A threshold set so that at most 5% of clean answers to separate calibration articles alarm falls among the least confident clean answers, and every more confident hijack passes. In our main gate run on the three API-served systems, the monitor misses 51-70% of the hijacks caused by a plain injected note (per-system point estimates). When the note adds one handwritten sentence that calls the target certain and asks for full probability, the monitor misses 99-100% of the hijacks, although low confidence still ranks them at AUROC 0.70-0.79. Confidence monitors should be tested at their operating alarm rate, against attackers who claim certainty.
| Subjects: | Computation and Language (cs.CL); Cryptography and Security (cs.CR) |
| Cite as: | arXiv:2611.02522 [cs.CL] |
| (or arXiv:2611.02522v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.02522 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zike Yang [view email]
[v1]
Mon, 2 Nov 2026 04:49:25 UTC (98 KB)
来源：arXiv cs.CL · arxiv.org