arXiv cs.CL· Yoshifumi Kawasaki·· 13 小时前AI 评分23
研究评估 10 款 LLM 在 21 个西班牙语国家的地区词汇知识
Evaluating Geographic Lexical Knowledge in Large Language Models Across 21 Spanish-Speaking Countries
AI 导读
一项研究用覆盖 21 个西班牙语国家、934 个词汇条目的专家数据库，将 10 款专有与开源权重 LLM 当作虚拟受访者，测试其按国家判断词汇使用的能力。评测包含 10,164 道 Yes-No 题与 16,637 道多选题，结果显示模型、国家和题型间差异显著：西班牙在 8 款模型的 Yes-No 题和 9 款模型的多选题中得分最高，智利在所有模型的多选题中得分最低。
正文 · 原文
Abstract:Large language models (LLMs) increasingly serve users across regional varieties, yet their knowledge of geographic lexical variation remains insufficiently evaluated. We investigate country-conditioned lexical usage judgments in Spanish using an expert-curated database covering 934 lexical items across 21 Spanish-speaking countries. Treating ten proprietary and open-weight LLMs as virtual informants, we evaluate two complementary question formats: Yes-No question format (YNQF) and multiple-choice question format (MCQF). The evaluation comprises 10,164 YNQF questions and 16,637 MCQF questions, assessed using F1 and an adjusted Jaccard coefficient, respectively. Results reveal substantial differences across models, countries, and formats. Spain receives the highest country-level score for eight models in YNQF and nine in MCQF; Chile receives the lowest MCQF score for every model. Regression analyses suggest that digital resource volume is generally more strongly associated with YNQF performance, whereas lexical distance from Spain is more strongly associated with MCQF performance. These findings provide a fine-grained diagnostic of uneven regional lexical knowledge and highlight the importance of complementary elicitation formats and evaluation metrics. The benchmark measures explicit usage judgments, motivating separate assessment of region-appropriate generation.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2602.09346 [cs.CL] |
| (or arXiv:2602.09346v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2602.09346 arXiv-issued DOI via DataCite |
Submission history
From: Yoshifumi Kawasaki [view email]
[v1]
Tue, 10 Feb 2026 02:42:22 UTC (648 KB)
[v2]
Tue, 27 Oct 2026 13:42:08 UTC (4,584 KB)
来源：arXiv cs.CL · arxiv.org