arXiv cs.CL· Ahmad Samuel Gali (Independent Researcher), Tadesse Destaw Belay (Instituto Polit\'ecnico Nacional), Idris Abdulmumin (University of Pretoria), Seid Muhie Yimam (University of Hamburg), Shamsuddeen Hassan Muhammad (Bayero University Kano, Imperial College London)·· 9 小时前AI 评分31
AfriNORM:评估 LLM 非洲文化规范推理能力的基准
AfriNORM: An Evaluation Benchmark for Normative Reasoning of LLMs Across African Cultures
AI 导读
研究者提出 AfriNORM,一个基于场景的选择题数据集,用于测量和诊断 LLM 在非洲社会文化语境下的规范推理局限。该基准引入 Universal Commonsense Bias Score(UCBS)和 Distractor Bias Score(DBS)两项指标,分别量化模型的常识偏见与对非洲文化规范的理解不足。
正文 · 原文
Abstract:Large language models (LLMs) are increasingly deployed across various sectors. However, current evaluation benchmarks are insufficient to assess the applicability of these models in local African contexts, as most focus on linguistic fluency (grammar/translation) rather than cultural competence. The few existing benchmarks that measure general knowledge or social conventions do not capture the African deontic modality, i.e., the strict norms and taboos that govern diverse social interactions. In this paper, we introduce AfriNORM, a scenario-based, multiple-choice question dataset designed to measure and diagnose the normative reasoning limitations of LLMs when applied to African sociocultural contexts. We introduce two metrics: a Universal Commonsense Bias Score (UCBS) to quantify commonsense bias in LLM and a Distractor Bias Score (DBS) to quantify an LLM's limited understanding of African cultural norms. We also conduct a case study on the Yorùbá language, identifying confusion gaps in two closed-source LLMs, "Gemini 2.0 Flash" and "GPT-5", and one open-source LLM, "Llama 3.1 8B Instruct" when prompted in English versus Yorùbá. While their accuracy in English is high, their performance in Yorùbá is much lower, attributable to cultural confusion rather than simple bias, indicating that the models exhibit semantic confusion when reasoning about norms in Yorùbá.
| Comments: | 8 pages, 3 figures, 5 tables |
| Subjects: | Computation and Language (cs.CL) |
| ACM classes: | I.2.7 |
| Cite as: | arXiv:2611.22580 [cs.CL] |
| (or arXiv:2611.22580v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22580 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ahmad Gali [view email]
[v1]
Tue, 17 Nov 2026 11:04:36 UTC (454 KB)
来源:arXiv cs.CL · arxiv.org