arXiv cs.CL· Manan Uppadhyay, Himanshu Beniwal, Prashant Kodali, Sunayana Sitaram·· 13 小时前AI 评分34
多语言基准审计：51 个基准、242 个数据集、219 种语言，覆盖广而不均
The State and Fate of Multilingual, Contextual Evaluation in the NLP World
AI 导读
一项数据驱动的审计分析了 51 个近期多语言基准、242 个数据集和 219 种语言，发现 36% 的语言仅出现在单一基准中，大洋洲、美洲、中亚等地区接近零覆盖。低资源语言仅被评估 1-3 类任务，而高资源语言达 14 类；56% 的数据集-语言实例由英文翻译构建，并存在翻译基准泄漏与平行语料重叠等污染问题。
正文 · 原文
Abstract:Multilingual evaluation benchmarks are the primary instrument for assessing whether large language models generalize beyond English, yet the adequacy of these benchmarks has received little systematic scrutiny. We present a data-driven audit of 51 recent multilingual benchmarks spanning 242 datasets and 219 languages, organized around three pillars: coverage, representativeness, and rigor. Our analysis reveals that coverage is wide but thin with 36% of evaluated languages appearing in only a single benchmark, entire regions (Oceania, the Americas, Central Asia) are near-zero, and a stark task equity gap leaves low-resource languages evaluated on only 1-3 task categories versus 14 for high-resource languages. Representativeness is structurally compromised: translation from English remains the dominant construction strategy where 56% of all dataset-language instances are translated introducing artifacts and English-centric framing, while culturally grounded content is concentrated in a handful of community-driven benchmarks with narrow language scope. The ecosystem thus forces a trade-off between breadth and validity. Rigor is undermined by benchmark contamination, including translated benchmark leakage and parallel corpus overlap that evade surface-form detection. We synthesize these findings into concrete recommendations for building evaluation frameworks that are natively constructed, culturally grounded, contamination-aware, and designed to serve the communities whose languages they claim to evaluate. The dashboard with all the metadata collected can be found at this https URL.
| Comments: | Published at Conference on Language Modelling, 2026. Can be found at this https URL |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.36990 [cs.CL] |
| (or arXiv:2610.36990v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36990 arXiv-issued DOI via DataCite (pending registration) |
|
| Journal reference: | Conf. on Language Modeling (COLM), 2026 |
Submission history
From: Manan Uppadhyay [view email]
[v1]
Tue, 27 Oct 2026 10:37:01 UTC (12,288 KB)
来源：arXiv cs.CL · arxiv.org