arXiv cs.CL· Ayana Niwa, Miyu Yamada, Kouta Nakayama, Masahiro Kaneko, Hisami Suzuki, Satoshi Sekine·· 13 小时前AI 评分31
重新审视 LLM 事实核查器的评估方法
Rethinking the Evaluation of LLM Fact-Checkers
AI 导读
研究提出 JSFactCheckBench，一个明确指定问题范围、判定条件和证据来源三项设计决策的日文基准，包含 1913 条来自 LLM 回答的声明，均针对日本社交媒体流传的误导信息。
正文 · 原文
Abstract:Large language models (LLMs) are widely used to answer factual questions, but their answers can contain errors and must be fact-checked claim by claim. Yet the resulting score depends on three design decisions: which questions the claims answer, under what conditions each claim is judged, and which sources count as evidence. No existing benchmark specifies all three. We introduce JSFactCheckBench, a Japanese benchmark that specifies all three, with 1,913 claims from LLM answers to questions about misleading information that circulated on Japanese social media, each annotated for factuality at a reference time and for importance to the question, using evidence from preselected Web sources. Across six LLMs, all verify claims directly answering the question less accurately than supplementary claims, a gap that aggregate scores obscure. Scores also drop when models must locate the relevant content in longer or multiple sources and, for all four open models, when the surrounding question and response are added. These weaknesses are visible only because the benchmark records the conditions behind each label. A benchmark that leaves them implicit cannot reveal them.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.30504 [cs.CL] |
| (or arXiv:2610.30504v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.30504 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ayana Niwa [view email]
[v1]
Wed, 21 Oct 2026 22:36:44 UTC (531 KB)
来源：arXiv cs.CL · arxiv.org