arXiv cs.CY· Christian Pipal, Morgan Wack, Eva-Maria Vogel·· 11 小时前AI 评分31
谁被统计在内？大语言模型、信息性拒答与姓名人口统计分类中的下游偏差
Who Gets Counted? Large Language Models, Informative Refusal, and Downstream Bias in Name-Based Demographic Classification
AI 导读
研究对比 3 个 LLM 与 7 种传统工具在姓名性别、种族、族裔推断上的表现，发现 LLM 的准确率优势部分来自选择性标注：Claude Haiku 只标注了一半选民，而传统工具仅取其置信度最高的一半时反而更准。
正文 · 原文
Abstract:Researchers routinely infer gender, race, and ethnicity from names, yet the standard tools used for this task misclassify the groups that diversity research most needs to measure. Large language models (LLMs) have been proposed as a remedy because they label diverse names accurately. We ask whether that accuracy carries over to the estimates researchers report. Across 10,000 Chinese names, 19,998 North Carolina voters with self-reported race, and 300 political science authors with human reference codes, we compare three LLMs with seven traditional tools and link the voter sample to party registration and age. Selective labeling explains part of the LLMs' accuracy advantage. Claude Haiku labels half of the voters, and a traditional tool limited to its most confident half is more accurate. Hard labels from every tool understate the White--Black gap in Democratic registration by 36% to 65%, while BIRDiE estimation on BISG posteriors comes within 0.03 of it. Haiku's missing labels select on age, inflating the age gap by six years. Combined with BISG, LLM answers sharpen individual labels without a demonstrated gain in BIRDiE accuracy. Classifiers should be judged by the estimates they produce, including the consequences of missing labels
| Subjects: | Computers and Society (cs.CY) |
| Cite as: | arXiv:2611.06550 [cs.CY] |
| (or arXiv:2611.06550v1 [cs.CY] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06550 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Christian Pipal [view email]
[v1]
Wed, 4 Nov 2026 14:15:04 UTC (307 KB)
来源：arXiv cs.CY · arxiv.org