arXiv cs.CL· Blessing Ngozi Uzor, Valdini Douglace Lemofouet, Paula Chikaodinaka Anyanwu, Tassallah Abdullahi, Clemencia Siro, Idris Abdulmumin, Seid Muhie Yimam, Shamsuddeen Hassan Muhammad·· 12 小时前AI 评分31
语言模型真的理解伊博文化吗？CAB-Igbo 基准揭示语言流利与文化知识的脱节
Do Language Models Understand Igbo Culture? Beyond Multiple Choice
AI 导读
研究者发布 CAB-Igbo 基准，基于伊博本土文化资料构建、经母语者核验，覆盖六个领域，用多项选择准确率和人工评估（Epistemic Fidelity、Representational Richness、Linguistic Quality、Pragmatic Proficiency）测试多款模型。
正文
Abstract:The value of language models for any community depends on whether a model actually understands the culture of the people it intends to serve. We use the Igbo language as a case study. Existing benchmarks test translated Western ideas, lacking depth. To address this, we introduce CAB-Igbo, a benchmark developed from indigenous cultural sources and verified by native speakers across six domains. We evaluated diverse models under varying conditions, incorporating regional personas. Performance was assessed using multiple-choice accuracy and human evaluations across Epistemic Fidelity, Representational Richness, Linguistic Quality, and Pragmatic Proficiency. Results reveal a stark dissociation between language fluency and cultural knowledge. Through statistical testing, we show that for most models, the severe lack of Epistemic Fidelity and Representational Richness is already present in English and does not significantly worsen when switching to Igbo. Instead, native language prompting mainly degrades Linguistic Quality and Pragmatic Proficiency. Gemini 2.0 Flash is the only model whose cultural knowledge scores improve under Igbo prompting. We find that current models struggle to demonstrate the knowledge needed for true alignment with Igbo, and multiple-choice formats allow pattern-matching to appear as understanding. Furthermore, automated judges agree with humans on linguistic quality but fail on epistemic fidelity. Ultimately, cultural evaluation must be built from community knowledge, establishing human evaluation as the gold standard to prevent misrepresenting low-resource communities globally.
| Comments: | 20 pages, 7 figures, 14 tables |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.35168 [cs.CL] |
| (or arXiv:2610.35168v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35168 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Blessing Ngozi Uzor [view email]
[v1]
Mon, 26 Oct 2026 10:31:11 UTC (161 KB)
来源：arXiv cs.CL · arxiv.org