arXiv cs.CL· Alexi Canesse, Mathis Le Bail, Ma\"el Jenny, Cl\'ement Elliker, Mahammed El Sharkawy, Sonia Vanier·· 10 天前AI 评分27
PatchBench：衡量激活修补附带损害的基准
PatchBench: Measuring Collateral Damage in Activation Patching
AI 导读
研究者推出 PatchBench，一个由 400 个高置信度越狱失败样本构成的基准，用于衡量 LLM 安全修补的附带损害。其评估协议 PatchBench-Local 为每个有害提示生成有害变体、结构匹配的良性提示和复用有害关键词的良性提示，以区分选择性修复与局部抑制。用该协议加 MMLU 评估四种激活引导方法发现，全局能力几乎不变时局部良性回归却很严重。
正文 · 原文
Abstract:An LLM safety patch can pass a benchmark while still being a poor repair. This risk is especially acute for jailbreak repairs, where the goal is to correct a specific unsafe behaviour without changing unrelated behaviours. A patch may block exact evaluation prompts yet fail on close harmful variants, or suppress harmful behaviour by over-refusing benign prompts that share its wording or structure. Existing protocols primarily test whether models can be broken, while aggregate metrics (attack success, refusal rates, global capability) cannot distinguish selective repairs from broader local suppression. To address this gap, we introduce PatchBench, a benchmark of empirically observed model-specific jailbreak failures inducing actionable harmful answers. Starting from 27,870 prompts from 37 public datasets, we curate 15,314 English prompts and query 8 open-source instruction-tuned models. Combining WildGuard filtering, pairwise Elo ranking, and manual verification, we retain a curated bank of 400 high-confidence jailbreak failures. We further introduce PatchBench-Local, an evaluation protocol testing whether a patch is behaviourally precise. For each harmful source prompt, PatchBench-Local generates three families of local neighbours: harmful variants preserving malicious intent, benign prompts with matched structure, and benign prompts reusing key harmful terms. It evaluates harmful-neighbour correction and benign-neighbour preservation, distinguishing selective repair from broader local suppression. Evaluating four activation steering methods with PatchBench-Local and MMLU shows that global capability can remain nearly unchanged while local benign regressions are severe, confirming aggregate metrics miss important collateral damage. PatchBench-Local provides a more precise basis for developing and comparing jailbreak repair methods.
| Comments: | Accepted to NeurIPS 2026 (Datasets and Benchmarks Track) |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL); Cryptography and Security (cs.CR) |
| Cite as: | arXiv:2610.10276 [cs.LG] |
| (or arXiv:2610.10276v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.10276 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mahammed El Sharkawy [view email]
[v1]
Wed, 7 Oct 2026 15:43:28 UTC (54 KB)
来源：arXiv cs.CL · arxiv.org