arXiv cs.CL· Dingjie Song, Zhiling Yan, Ziqian Zhang, Yuxuan Zhang, Lichao Sun·· 2 天前AI 评分36
PremiseCheck：高效核查人类与 AI 同行评审中的事实错误
PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews
AI 导读
研究者提出 ReviewFact 基准，在 NLP、机器学习与自然科学领域用统一定义标注评审意见中的事实错误，并同时衡量成本与延迟。PremiseCheck 框架让 LLM 在不读论文的情况下把每条评审意见改写为显式前提，再由决策模型逐条对照论文核查。单条核查时，其在三个领域准确率与 GPT-5.5 相当而成本约为一半，NLP 评审上成本降至四分之一、延迟不到一半。
正文 · 原文
Abstract:Peer review is under strain from the growth of submissions, and review comments written by humans or AI can misstate what a paper says. We introduce ReviewFact, a benchmark that labels such factual errors in review comments under one definition across NLP, machine learning, and the natural sciences, and that measures cost and latency alongside accuracy. A frontier LLM checks comments well but must read the whole paper for each check, while cheap decision models, which score whether a statement holds in a text, fall far short on whole comments that mix several implicit assertions. PremiseCheck is a framework that combines the two. An LLM rewrites each comment as explicit premises without reading the paper, and a decision model checks each premise against the paper. When each comment is checked on its own, PremiseCheck reaches accuracy comparable to GPT-5.5 across all three fields at about half its cost, and on NLP reviews at a quarter of its cost and less than half its latency.
| Comments: | 29 pages, 19 figures, 23 tables. Code: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.21046 [cs.CL] |
| (or arXiv:2610.21046v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21046 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhiling Yan [view email]
[v1]
Thu, 15 Oct 2026 00:42:49 UTC (725 KB)
来源：arXiv cs.CL · arxiv.org