arXiv cs.CL· Kihyun Na, Injung Kim·· 12 小时前AI 评分31
ReviewMap：区分同行评审评论中的评估行为与自动化边界
ReviewMap: Mapping Evaluative Acts and Automation Boundaries in Peer-Review Comments
AI 导读
ReviewMap 框架将同行评审评论分为两类评估行为：对论文主张与学术标准的评估，以及对贡献价值与会议匹配度的校准，并按行为类型条件性应用细粒度属性。在 500 篇 ICLR 2026 论文上经多模型 LLM 共识验证，行为层在模型间 κ ≥ 0.75 的操作准则下可可靠自动化，属性层则需人工监督，由此划定自动化边界。
正文 · 原文
Abstract:Peer-review comments check what a paper does and judge why it matters, but no existing scheme distinguishes these two evaluative acts. We argue that making these acts explicit and measuring how reliably LLMs identify them is a prerequisite for responsible LLM-assisted review analysis: humans need to inspect review judgments and their grounds. This defines a more honest mission for NLP in peer review than full automation. We present ReviewMap, a framework that classifies each comment as assessment of claims and scholarly standards or calibration of contribution value and venue fit, with finer-grained attributes applied conditionally by act type. Applied to 500 ICLR~2026 papers through multi-model LLM consensus, the act layer proves reliably automatable under the operational guideline (inter-model $\kappa \geq 0.75$), supported by five-human validation on 470 development comments, while the attribute layer requires human oversight, defining a measured automation boundary. At the corpus level, 17% of reviewer-paper instances contain no panel-coded calibration, and the panel-coded calibration share is nearly $3\times$ as high in Strengths as in Weaknesses. This boundary informs three affordances: a review guideline that operationalizes the assessment-then-calibration workflow for reviewers, a per-comment rebuttal strategy scaffold for authors, and a discussion-phase dashboard for area chairs.
| Comments: | 27 pages, 5 figures, 19 tables |
| Subjects: | Computation and Language (cs.CL); Human-Computer Interaction (cs.HC) |
| Cite as: | arXiv:2611.07489 [cs.CL] |
| (or arXiv:2611.07489v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07489 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Kihyun Na [view email]
[v1]
Thu, 5 Nov 2026 05:49:01 UTC (162 KB)
来源：arXiv cs.CL · arxiv.org