arXiv cs.AI· Weitong Li, Donglin Song, Junkai Lin, Xi Xuan·· 3 天前AI 评分23
论文指出 BBQ 偏见分数混淆可答性，提出 GCD 方向设计
Beyond Bias Scores: Answerability Confounds in Activation Steering with Gradient-Constrained Direction Design (GCD)
AI 导读
研究在 BBQ 基准上发现偏见分数混淆了模型偏好与是否作答，可答性通道在 99 种配置中有 17% 的情况贡献超过一半分数变化、11% 的情况贡献全部变化。作者提出 Gradient-Constrained Direction Design（GCD），在 Qwen-7B 协议下将作答率失真降低 14 倍、偏好改变扩大 4 倍，并建议在报告分数时同时披露可答性。
正文 · 原文
Abstract:Bias scores in language-model evaluations often conflate what a model prefers with whether it answers at all. We identify this confound in \textsc{BBQ}, whose ambiguous score factorizes into answerability and signed preference. Across 11 categories, 28,}824 items, and five models, contrastive steering lowers the signed ambiguous score in 11/11 categories but lowers the signed disambiguated preference score in only 8/11. Across 99 configurations, the answerability channel exceeds half of the observed score change in 17\% and accounts for the entire change in 11%. In a six-category disambiguated evaluation, 20.8\% of 1,500 items flip to the unknown option, where abstention is incorrect; \textsc{StereoSet} shows no analogous answer-rate shift. The effect reflects selection and near-orthogonality between the contrastive and preference-gradient directions. We introduce Gradient-Constrained Direction Design (\textsc{GCD}), an option-gradient construction with a first-order unknown-logit constraint. Under a matched Qwen-7B protocol, GCD reduces answer-rate distortion 14-fold and yields a 4-fold larger preference change, while ablations show similar gains for alternative and unconstrained option-gradient directions; nulling the unknown gradient alone is not responsible for the gain. We recommend reporting answerability alongside published scores.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.17641 [cs.AI] |
| (or arXiv:2610.17641v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17641 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Junkai Lin [view email]
[v1]
Tue, 13 Oct 2026 03:09:29 UTC (2,787 KB)
来源：arXiv cs.AI · arxiv.org