arXiv cs.CL· Yaochen Wang, Chenguang Wang, Ming Li, Haomin Zhuang, Xinyue Zeng, Han Chen, Kwesi Adu Cobbina, Dang Nguyen, Dianqi Li, Dawei Zhou, Tianyi Zhou·· 11 小时前AI 评分38
AI 审稿人能否识别论文修改的改进？基于 PaperTrajectory-2026 的大规模研究
Can AI Reviewers See Your Efforts to Improve Your Paper? A Large-Scale Study of Paper Revision Trajectories
AI 导读
研究基于 PaperTrajectory-2026 数据集（涵盖五个 AI 会议的 6,959 条论文修改轨迹）发现，八个 AI 审稿人对大多数重大改进给出的分数保持不变，且对分数变化方向的判断一致性很弱。即使修改获得更高的中性评分，一条简短的负面用户意见也能抹去或逆转这一优势，且不同审稿人之间差异显著。
正文 · 原文
Authors:Yaochen Wang, Chenguang Wang, Ming Li, Haomin Zhuang, Xinyue Zeng, Han Chen, Kwesi Adu Cobbina, Dang Nguyen, Dianqi Li, Dawei Zhou, Tianyi Zhou
Abstract:AI reviewers are increasingly used to assess scientific manuscripts, yet it remains unclear whether their judgments reflect the improvements authors make through revision. We investigate this question using PaperTrajectory-2026, the 2026 release of PaperTrajectory, a dataset of 6,959 paper trajectories across five AI conferences, with automatic annotations of changes along 14 dimensions and separate judgments of overall quality. The annotations show that major improvements emphasize empirical evidence and framing, while later revisions tend toward incremental improvement. Using independent assessments of manuscript versions from eight AI reviewers, we examine whether annotated major improvements receive higher scores, which revision dimensions are associated with score changes, and whether user opinions can outweigh observed revision gains. Every reviewer leaves scores unchanged on most major improvements, and agreement on the direction of score changes is weak. Revision dimensions show some reviewer-specific associations with score increases, but provide limited predictive discrimination and no consistent guide across reviewers. Even when a revision earns a higher neutral score, a brief negative user opinion can erase or reverse that advantage, with substantial variation across reviewers. These findings show that substantive revisions are not consistently reflected in review scores and that observed scoring gains can be vulnerable to user framing, motivating assessments that connect manuscript evidence to overall judgments and remain stable when no scientific evidence changes.
| Comments: | 43 pages including appendices, 8 figures |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.02244 [cs.CL] |
| (or arXiv:2611.02244v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.02244 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Chenguang Wang [view email]
[v1]
Sun, 1 Nov 2026 22:06:26 UTC (1,875 KB)
来源：arXiv cs.CL · arxiv.org