arXiv cs.CL· Sarah Almeida Carneiro (X), Christos Xypolopoulos (X, NTUA), Xiao Fei (X), Yang Zhang (X), Michalis Vazirgiannis (X, MBZUAI)·· 13 小时前AI 评分22
CARTE：评估大语言模型对法国各地区知识掌握程度的基准
CARTE: A Benchmark for Mapping Language Model Knowledge Across France
AI 导读
研究者推出 CARTE 基准，用 3,132 道多选题覆盖法国 13 个大区、12 个主题领域，评估大语言模型对法国境内区域差异化知识的辨别能力。该基准还包含聚焦法语地区语言差异的子集 CARTE-LV，题目经人工审核、LLM-as-a-judge、模型难度校准和二次人工审核四阶段验证。
正文 · 原文
Abstract:We introduce CARTE (Culturally Anchored Regional-Territorial Evaluation), a multiple-choice benchmark for evaluating the ability of large language models (LLMs) to discriminate fine grained, geographically grounded and regionally differentiated knowledge within France. While prior benchmarks focus on national-level cultural understanding, they largely overlook intra-country variation and the need to distinguish between closely related regional contexts. CARTE addresses this gap by introducing 3,132 questions spanning the 13 metropolitan regions of France and covering 12 thematic domains, including culture, language, demographics, economy, environment, and mobility. Questions underwent four-stage validation: human review, LLM-as-a-judge, model-based difficulty calibration, and a second human review. We further introduce CARTE-LV, a subset targeting Linguistic Variation across French regions, enabling focused evaluation of language-related differences. We evaluate 32 open-weight LLMs of 1B to 30B parameters and three frontier API models across zero-, one-, and three-shot settings. Our experiments reveal performance disparities across regions and model scales, suggesting systematic gaps in pretraining coverage and limited robustness to intra-national variation.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2606.01995 [cs.CL] |
| (or arXiv:2606.01995v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.01995 arXiv-issued DOI via DataCite |
Submission history
From: Sarah Almeida Carneiro [view email] [via CCSD proxy]
[v1]
Mon, 1 Jun 2026 09:50:50 UTC (191 KB)
[v2]
Thu, 22 Oct 2026 14:58:05 UTC (4,121 KB)
来源：arXiv cs.CL · arxiv.org