arXiv cs.AI· Kiwoong Park, Seyoung Song, Donghyun Woo, Alice Oh·· 3 天前AI 评分27
北还是南？诊断 LLM 对朝鲜语言、社会与知识的理解
North or South? Diagnosing LLMs' Understanding of North Korean Language, Society, and Knowledge
AI 导读
研究团队发布 LENS 基准，用 1,765 道人工校验题目从语言鲁棒性、社会文化视角对齐和领域知识三个维度评测 LLM 对朝鲜的理解。评测显示，模型在朝鲜语输入上性能小幅下降，社会文化视角对齐差距更大，领域知识则严重不足。定性分析发现模型常混淆正式制度与现实生活、套用自由民主假设或依赖刻板印象。
正文 · 原文
Abstract:Large Language Models (LLMs) have made substantial progress in multilingual question answering, yet their ability to understand low-resource sociolinguistic contexts remains under-explored. We study North Korea as a unique testbed where decades of division have produced distinct differences from South Korea in vocabulary, orthography, norms, and ideological discourse. We introduce LENS (Language and sociocultural Evaluation for North Korean Society), a human-validated benchmark of 1,765 questions evaluating LLMs across three dimensions: linguistic robustness, sociocultural perspective alignment, and domain-specific knowledge. Evaluating diverse recent LLMs, we find a modest performance drop on North Korean language inputs, a larger gap in sociocultural perspective alignment, and severe limitations on domain-specific knowledge. Qualitative analysis reveals that models often conflate formal institutions with lived realities, project liberal-democratic assumptions, or rely on coarse stereotypes. Our results demonstrate that linguistic similarity alone does not guarantee sociocultural or domain-specific understanding, highlighting the critical need for grounded evaluation resources for underrepresented cultural and political contexts.
| Comments: | Accepted to AACL-IJCNLP 2026 (Main Conference). 19 pages, 1 figure, 11 tables |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.15282 [cs.CL] |
| (or arXiv:2610.15282v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15282 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Kiwoong Park [view email]
[v1]
Sun, 11 Oct 2026 02:26:46 UTC (68 KB)
来源：arXiv cs.AI · arxiv.org