arXiv cs.CL· Alejandro Mozo Quesada (Universidad de M\'alaga, Spain), Sergio G\'alvez Rojas (Universidad de M\'alaga, Spain), Francisco L\'opez Valverde (Universidad de M\'alaga, Spain)·· 11 小时前AI 评分34
审计 System One 决策模型人口统计偏见的新协议：交换、复现、读取概率质量
Swap, Replicate, Read the Mass: A Protocol for Auditing Demographic Bias in System One Decision Models
AI 导读
研究提出一套针对 System One 决策模型的偏见审计协议，在 BBQ、CrowS-Pairs 和 SilverSpoon 三个基准上测试十二个模型加一个无思考消融，共 1,638,000 次决策。
正文 · 原文
Abstract:System One decision models replace text generation with typed, probability-scored answers and are entering production. Answers execute without a person in the loop, and no multi-model audit exists on established bias benchmarks. We provide that audit: three benchmarks: BBQ (all eleven categories), CrowS-Pairs, and SilverSpoon, adapted to the typed interface, twelve models plus a no-thinking ablation, and 1,638,000 decisions. Every order-sensitive item runs twice with options swapped; native typed models also swap labels; ten configurations repeat under extra seeds and batch 16. We find widespread stereotype alignment, order-dependent answers, and confidence that tracks instability rather than bias. In BBQ, 97 of 143 model-by-category scores are significant and 94 aligned, while the commercial reference abstains on 99.5% of ambiguous items, hiding aligned probability mass. Option swaps flip 0.0% to 72.6% of choices; typed scoring stays below 10% except Laya (22-33%), and untuned letter prompting reaches 73%. CrowS-Pairs preference spans 0.496 to 0.816, and boolean answers can be decided by position (1.00 versus 0.005). SilverSpoon judgments shift toward lower-SES labels in five of eight decision models, inherited from the 9B backbone or induced by tuning. Audits must permute options and reverse questions by default; order-invariant scoring should become a procurement requirement.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.00496 [cs.AI] |
| (or arXiv:2611.00496v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.00496 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Alejandro Mozo Quesada [view email]
[v1]
Sat, 31 Oct 2026 00:36:15 UTC (90 KB)
来源：arXiv cs.CL · arxiv.org