arXiv cs.CL· Viktoriia Makovska, Tetiana Zakharchenko, Oleksandra Ostafiichuk·· 10 小时前AI 评分26
WarBias：评估 LLM 对境内流离失所者与退伍军人刻板印象的乌克兰语-英语双语基准
WarBias: A Bilingual LLM Benchmark of Stereotypes About Internally Displaced People and Veterans
AI 导读
WarBias 是一个乌克兰语-英语双语基准，用于评估 LLM 对退伍军人和境内流离失所者（IDP）的刻板印象，包含 1,007 组“刻板印象-反刻板印象-无关”句子三元组及每种语言 240 道多选题。在六款模型的基线评测中，四款对至少一个群体的乌克兰语句子表现出刻板印象偏好，对 IDP 尤为明显，而英语得分大多均衡。该基准还提供成对的遗忘/保留材料，可用于基于遗忘的偏见缓解。
正文 · 原文
Abstract:This paper contains examples of stereotypes about veterans and displaced people that may be offensive or distressing. Social bias benchmarks for language models commonly examine categories such as gender, race, and religion, often drawn from lists of legally protected characteristics. Veterans and internally displaced people (IDPs) fall outside these lists, and stereotypes about them, which can undermine access to employment, housing, and social participation, remain largely absent from language model evaluation. We introduce WarBias, a Ukrainian-English benchmark of stereotypes about these two groups. Built from a taxonomy of 120 stereotype claims grounded in research and monitoring reports, WarBias contains 1,007 stereotype-counter-stereotype-unrelated sentence triplets and 240 multiple-choice questions per language. A Ukrainian language model selected in a blinded human pilot generated the Ukrainian triplets. At least two independent annotators reviewed each triplet in the annotation cohort. The authors revised items in response to the reviews, and the final release was machine-translated into English. Each claim carries a human severity rating. In a baseline evaluation of six models, four prefer stereotypical Ukrainian sentences about at least one group, most strongly about IDPs, while English scores are mostly balanced. WarBias supports complementary evaluations of stereotypical preferences and context-sensitive judgments, and provides paired forget/retain material for unlearning-based bias mitigation. We document the benchmark's construction, intended uses, and limitations, and offer its construction method as a template for analogous benchmarks in other conflict-affected societies.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.11284 [cs.CL] |
| (or arXiv:2611.11284v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.11284 arXiv-issued DOI via DataCite |
Submission history
From: Viktoriia Makovska [view email]
[v1]
Sun, 8 Nov 2026 10:12:11 UTC (1,336 KB)
[v2]
Tue, 10 Nov 2026 10:05:09 UTC (1,336 KB)
来源：arXiv cs.CL · arxiv.org