arXiv cs.AI· Jonghyeon Choi, Yeonjun Choi, Beakcheol Jang·· 1 天前AI 评分34
多测试国家偏见基准：为 LLM 设定显式无偏基线
Multi-Test Nation Bias Benchmarking of LLMs with Explicit Unbiased Baselines
AI 导读
研究用 722 份联合国安理会官方记录和五项互补测试（WOW、AK+-、PVS、F-Oppose、Headliner）量化 LLM 的国家层面偏见，并为每项测试显式定义无偏基线。对 GPT-4o-mini、Llama 3.3-70B、Mistral-22B-Small 和 Qwen3-30B 的基准测试显示一致模式：模型输出对中国和俄罗斯偏向反对（红）一侧，对英国、美国、法国偏向支持（蓝）一侧。
正文 · 原文
Abstract:While prior work has highlighted that systematic nation-level bias in large language models (LLMs) can pose operational risks for international relations (IR) applications, many existing evaluations still lack a clearly specified unbiased reference (ground truth), limiting fully quantitative and cross-setting measurement. Using an expanded corpus of 722 official UN Security Council (UNSC) records, we measure nation-level bias with five complementary tests, thereby extending prior approaches to cover a broader set of bias expression channels: opinion ranking (WOW), association sorting (AK+-), vote simulation (PVS), opposition attribution (F-Oppose), and framing (Headliner). To enable grounded bias evaluation, we explicitly define an unbiased status for each test; in particular, for WOW and AK+- where ground truth is difficult to specify, we propose an action-based pseudo unbiased status derived from observable voting behavior to construct a quantifiable baseline. Benchmarking 4 LLMs (GPT-4o-mini, Llama 3.3-70B, Mistral-22B-Small, and Qwen3-30B) across the five permanent members of the UNSC, we find a test-agnostic macro-pattern: relative to the reference baselines defined for each test, model outputs lean toward the opposition (red) side for China and Russia and toward the supportive (blue) side for the U.K., the U.S., and France. These results underscore the need to audit IR-facing LLM deployments with grounded, multi-test benchmarks tied to explicit reference points.
| Comments: | Published in Transactions of the Association for Computational Linguistics (TACL), 2026. 17 pages. Equal contribution: Jonghyeon Choi and Yeonjun Choi |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.41486 [cs.AI] |
| (or arXiv:2610.41486v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41486 arXiv-issued DOI via DataCite (pending registration) |
|
| Journal reference: | Transactions of the Association for Computational Linguistics, vol. 14, pp. 2374-2390, 2026 |
| Related DOI: | https://doi.org/10.1162/TACL.a.806
DOI(s) linking to related resources |
Submission history
From: Yeonjun Choi [view email]
[v1]
Fri, 30 Oct 2026 09:41:05 UTC (943 KB)
来源：arXiv cs.AI · arxiv.org