arXiv cs.CL· Zainab Afolabi, Nikita Kozodoi, Jack Butler, Babs Khalidson·· 12 小时前AI 评分38
NaijaFacts：区分尼日利亚事实问题中的语言效应与知识效应
NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria
AI 导读
研究者推出 NaijaFacts 基准，用尼日利亚母语者撰写、基于官方统计的事实与统计问题，提供英语、尼日利亚皮钦语、约鲁巴语、伊博语和豪萨语五个版本。在八个前沿及尼日利亚适配模型上，知识效应占主导：美英问题英语查询准确率 49–50%，尼日利亚仅 20%；语言效应较小且随网络代表性变化，英语到约鲁巴语在尼日利亚问题上降 9 分。
正文 · 原文
Abstract:Multilingual benchmarks for African languages mostly translate generic questions into local languages, measuring linguistic competence but not whether models know facts about Africa. We introduce \naija, a benchmark of factual and statistical questions about Nigeria, authored by native speakers and grounded in authoritative official statistics, with every question available in English, Nigerian Pidgin, \yoruba, Igbo, and Hausa. Answering similar questions for the UK and US yields a country by language matrix that disentangles a language effect (the model knows the fact but degrades in a low-resource language) from a knowledge effect (the fact is absent in every language). Across eight frontier and Nigeria-adapted models, the knowledge effect dominates: English-query accuracy is 49--50\% for the US/UK but 20\% for Nigeria. The language effect is smaller and tracks web representation, degrading from English to \yoruba{} (9 points on Nigeria, up to 24 on US/UK questions). We find scale does not help: the strongest models reach 66--79\% on the UK and the US but none exceed 31\% on Nigerian statistics. Extending to all 54 African countries with our best performing frontier model shows the same profile continent-wide, 96\% on culture and 98\% on politics against 26\% on statistics, so the gap is not specific to Nigeria.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.06504 [cs.CL] |
| (or arXiv:2611.06504v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06504 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zainab Afolabi [view email]
[v1]
Wed, 4 Nov 2026 13:34:35 UTC (1,858 KB)
来源：arXiv cs.CL · arxiv.org