arXiv cs.CL· Ayal Klein·· 12 小时前AI 评分22
可解码性不等于有效性：评估语言模型概念探针的干扰敏感性
Decodability Is Not Validity: Evaluating Nuisance Sensitivity in Language Model Concept Probes
AI 导读
两个概念探针在预测语言模型隐藏状态中的概念时准确率相当，但分数中目标概念与相关干扰属性的混合程度可能相差数倍。研究在 CEBaB、BiasBios 及亚马逊评论情感等四种目标-干扰设定、两个 LLM 上对比可解码性与判别证据，发现三者中有三种设定下留出 AUROC 统计等价的提取器其干扰敏感性相差可达数倍，且可解码性并非可靠代理指标。
正文 · 原文
Abstract:Two concept probes can be equally accurate at predicting a concept from a language model's hidden states while their scores contain different mixtures of the target concept and a correlated nuisance attribute. In four target--nuisance settings (two aspect sentiments in CEBaB; occupation and gender in BiasBios; Amazon review sentiment with negation, and with first-person framing), two LLMs, and a population of concept extractors, we compare decodability with discriminant evidence: the score's dependence on the nuisance at fixed target, and its response to input edits intended to change only the nuisance, filtered by two LLM judges and compared against placebo edits. Under correlated training data, extractors that are statistically equivalent in held-out AUROC differ up to several-fold in nuisance sensitivity in three of the four settings, and decodability is not a reliable proxy for it: their rank correlation changes sign across settings. The effect follows the probe's training data, not the nuisance's natural association with the target: first-person framing, unrelated to sentiment in natural reviews, is absorbed at least as much as negation, which co-occurs with negative sentiment, and decorrelated training removes the systematic effect. Selecting extractors with nuisance-aware development diagnostics, which need nuisance labels but no counterfactuals, lowers counterfactual nuisance sensitivity wherever the default choice is sensitive, at a small accuracy cost. Concept measurements should be validated, not just decoded.
| Comments: | 4 pages + appendix; code and data: this https URL |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| ACM classes: | I.2.7 |
| Cite as: | arXiv:2611.07805 [cs.CL] |
| (or arXiv:2611.07805v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07805 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ayal Klein [view email]
[v1]
Thu, 5 Nov 2026 10:47:17 UTC (158 KB)
来源：arXiv cs.CL · arxiv.org