跳到正文
arXiv cs.CL· Chenguang Wang, Ming Li, Chenrui Fan, Jianpeng Chen, Han Chen, Tianyi Zhou, Dawei Zhou·· 10 小时前AI 评分34

为可信 AI 审稿补上缺失一环:从修辞鲁棒性基准测试到 SciCore Review

A Missing Piece for Trustworthy AI Reviewers: From Benchmarking Rhetorical Robustness to SciCore Review

AI 导读

研究提出"修辞鲁棒性"概念,要求 AI 审稿人对内容相同但措辞不同的稿件给出一致评分,同时保持对不同论文的区分能力,并发布含 1,260 个稿件版本、覆盖 30 种审稿人配置的 RobustReview 基准。

正文 · 原文

View PDF HTML (experimental)

Abstract:AI reviewers can assign different judgments to manuscripts that report the same science in different wording, potentially rewarding rhetorical optimization over scientific improvement. We formulate Rhetorical Robustness as the joint requirement of stability across content-preserving rewrites and discrimination across papers. We introduce RobustReview, a controlled full-manuscript benchmark with 1,260 manuscript versions, and evaluate 30 reviewer configurations. The benchmark reveals false robustness, where low rewrite sensitivity coincides with score collapse across papers, and shows that human alignment and rhetorical robustness rank reviewers differently. Moreover, the evaluated content-focused prompting protocol does not consistently improve robustness across backbones. Motivated by these findings, we introduce SciCore, a dual-branch reviewer that averages a full-manuscript judgment with a judgment based on an extracted, structured science core. This design combines manuscript-level assessment with a content-normalized view intended to reduce rhetorical sensitivity. In our primary GPT-5.5 comparison, SciCore achieves a leading joint stability-discrimination profile among the benchmarked reviewers while maintaining competitive human alignment. These results identify rhetorical robustness as a distinct evaluation target and demonstrate the potential of science-core review to improve it.
Comments: 35 pages, 2 figures, 20 tables. Accepted (Oral) at AI-Native Academia @ NeurIPS 2026
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2609.39027 [cs.CL]
  (or arXiv:2609.39027v2 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2609.39027

arXiv-issued DOI via DataCite

Submission history

From: Chenguang Wang [view email]
[v1] Wed, 30 Sep 2026 05:32:26 UTC (1,195 KB)
[v2] Wed, 18 Nov 2026 19:47:58 UTC (1,195 KB)

来源:arXiv cs.CL · arxiv.org