arXiv cs.CL· Gregory N. Frank·· 12 小时前AI 评分58
研究称基于拒答的对齐评测失效，检测便宜而路由需学习
Detection Is Cheap, Routing Is Learned: Why Refusal-Based Alignment Evaluation Fails
AI 导读
论文指出当前对齐评测多衡量模型是否编码危险概念、是否拒答有害请求，却漏掉从概念检测到行为策略的路由层。作者以中文来源模型的政治审查为自然实验，对五个实验室的九个开放权重模型做探针、定向消融和行为测试，发现探针准确率本身不具诊断力，政治探针、空对照和置换基线都能达到 100%，只有留出类别的泛化才是有效检验。
正文 · 原文
Abstract:Current alignment evaluation mostly measures whether models encode dangerous concepts and whether they refuse harmful requests. Both miss the layer where alignment often operates: routing from concept detection to behavioral policy. We study political censorship in Chinese-origin language models as a natural experiment, using probes, surgical ablations, and behavioral tests across nine open-weight models from five labs. Three findings follow. First, probe accuracy alone is non-diagnostic: political probes, null controls, and permutation baselines can all reach 100%, so held-out category generalization is the informative test. Second, surgical ablation reveals lab-specific routing. Removing the political-sensitivity direction eliminates censorship and restores accurate factual output in most models tested, while one model confabulates because its architecture entangles factual knowledge with the censorship mechanism. Cross-model transfer fails, indicating that routing geometry is model- and lab-specific. Third, refusal is no longer the dominant censorship mechanism. Within one model family, hard refusal falls to zero while narrative steering rises to the maximum, making censorship invisible to refusal-only benchmarks. These results support a three-stage descriptive framework: detect, route, generate. Models often retain the relevant knowledge; alignment changes how that knowledge is expressed. Evaluations that audit only detection or refusal therefore miss the routing mechanism that most directly determines behavior.
| Comments: | Code and data: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2603.18280 [cs.LG] |
| (or arXiv:2603.18280v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2603.18280 arXiv-issued DOI via DataCite |
Submission history
From: Gregory Frank [view email]
[v1]
Wed, 18 Mar 2026 20:54:34 UTC (883 KB)
[v2]
Sun, 12 Apr 2026 00:49:35 UTC (883 KB)
[v3]
Fri, 1 May 2026 17:19:25 UTC (887 KB)
[v4]
Tue, 20 Oct 2026 12:05:42 UTC (887 KB)
来源：arXiv cs.CL · arxiv.org