arXiv cs.CL· Seoyeon Kim, Jeonghyun Kang, Hamin Koo, Barom Kang, Heondeuk Lee, Jaehyung Kim·· 11 小时前AI 评分22
KLEAR：通过受控基准变换实现按应用需求匹配知识的韩语 LLM 评测
KLEAR: Knowledge-Matched LLM Evaluation in Korean across Application Requirements via Controlled Benchmark Transformation
AI 导读
KLEAR 是一个韩语 LLM 评测框架，通过保持所需知识不变、只改变应用需求来构建知识匹配的对照题目，从而区分模型答错是因为缺少知识还是不会应用知识。分析显示现有韩语基准中多数题目仅靠事实回忆即可作答，而要求更高应用的题目所需知识也不同，难以单独衡量应用能力。实验表明生成的知识匹配题集基本满足约束，随着应用需求提高，即使提供所需知识，模型仍会暴露持续性失败。
正文 · 原文
Abstract:Existing Korean benchmarks for Large Language Models (LLMs) evaluate performance across diverse knowledge domains, but their accuracy scores cannot distinguish whether incorrect responses arise from a lack of required knowledge or a failure to apply that knowledge correctly. Our analysis of existing Korean benchmarks reveals that most questions can be answered through factual recall alone, while those demanding greater application also differ in required knowledge, making it difficult to isolate application ability. To address this, we introduce KLEAR, a framework that constructs knowledge-matched contrasts by varying application demand while preserving the required knowledge. During construction, KLEAR identifies the knowledge underlying each source question as an explicit anchor bundle to guide generation of questions requiring more complex application. Generated candidates then undergo automated and human validation, with iterative revision based on evaluation feedback. Experiments demonstrate that the resulting knowledge-matched sets largely satisfy the intended constraints, enabling controlled evaluation across diverse LLMs. These evaluations reveal that increasing application demand exposes failures that persist even when the required knowledge is provided.
| Comments: | 34 pages, under review |
| Subjects: | Computation and Language (cs.CL) |
| ACM classes: | I.2.7 |
| Cite as: | arXiv:2611.09223 [cs.CL] |
| (or arXiv:2611.09223v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.09223 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Seoyeon Kim [view email]
[v1]
Fri, 6 Nov 2026 09:42:50 UTC (557 KB)
来源：arXiv cs.CL · arxiv.org