arXiv cs.CL· Mengsay Loem, Kentaro Imajo, Ryota Ozaki, Masanori Hirano·· 11 小时前AI 评分20
Khmer PFGen-Bench：面向低资源语言高棉语开放生成的无评判基准
Khmer PFGen-Bench: Building a Judge-Free Benchmark for Open-Ended Generation in a Low-Resource Language
AI 导读
Khmer PFGen-Bench 是首个针对高棉语多句开放问答的无 LLM 评判基准，改编自日语的 PFGen-Bench，用关键词规则和 n-gram 统计对 LLM 生成的参考答案打分。研究发现后训练模型给出的参考更有效但多样性更低，混合不同模型家族可在相同参考数量下提升多样性；使用九个家族的后训练模型参考时，模型排名在不同参考集间稳定，且与母语者评分一致。
正文 · 原文
Abstract:Benchmarks for open-ended generation often rely on large language model (LLM) judges, whose reliability in low-resource languages is uncertain. We introduce Khmer PFGen-Bench, to our knowledge the first judge-free benchmark for multi-sentence answers to questions composed directly in Khmer, a low-resource language. Khmer PFGen-Bench adapts the scorer of PFGen-Bench, a benchmark for Japanese that scores responses with keyword rules and n-gram statistics from LLM-generated references. This scoring requires references that are both valid and varied, which LLMs may not produce in Khmer, so we examine how to build such references. We find that post-trained models can give more valid but less varied references than their base variants, and that mixing model families increases variety for the same number of references. Using references from post-trained models in nine families, the benchmark yields model rankings that are stable across reference sets and consistent with a native speaker's ratings. These findings offer guidance for building judge-free benchmarks in other low-resource languages.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.09222 [cs.CL] |
| (or arXiv:2611.09222v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.09222 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mengsay Loem [view email]
[v1]
Fri, 6 Nov 2026 09:39:50 UTC (650 KB)
来源：arXiv cs.CL · arxiv.org