arXiv cs.AI· Xiaonan Wang, Jinyoung Yeo, Joon-Ho Lim, Hansaem Kim·· 2 天前AI 评分20
KULTURE Bench：评估语言模型韩语文化语境理解能力的基准
KULTURE Bench: A Benchmark for Assessing Language Model in Korean Cultural Context
AI 导读
KULTURE Bench 是一个专为韩国文化设计的评测框架，包含文化新闻、成语和诗歌数据集，从词、句、段落三个层面评估语言模型的文化理解与推理能力。研究团队用该基准测试了不同语料训练的模型，结果显示模型对韩国文化深层文本的理解仍有较大提升空间。
正文 · 原文
Abstract:Large language models have exhibited significant enhancements in performance across various tasks. However, the complexity of their evaluation increases as these models generate more fluent and coherent content. Current multilingual benchmarks often use translated English versions, which may incorporate Western cultural biases that do not accurately assess other languages and cultures. To address this research gap, we introduce KULTURE Bench, an evaluation framework specifically designed for Korean culture that features datasets of cultural news, idioms, and poetry. It is designed to assess language models' cultural comprehension and reasoning capabilities at the word, sentence, and paragraph levels. Using the KULTURE Bench, we assessed the capabilities of models trained with different language corpora and analyzed the results comprehensively. The results show that there is still significant room for improvement in the models' understanding of texts related to the deeper aspects of Korean culture.
| Comments: | Accepted by the 38th Pacific Asia Conference on Language, Information and Computation |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2412.07251 [cs.CL] |
| (or arXiv:2412.07251v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2412.07251 arXiv-issued DOI via DataCite |
Submission history
From: Xiaonan Wang [view email]
[v1]
Tue, 10 Dec 2024 07:20:51 UTC (1,439 KB)
来源：arXiv cs.AI · arxiv.org