arXiv cs.CL· Fan Huang, Hongyu Zhou·· 11 小时前AI 评分38
基于 LLM 的学术论文质量评估评审范式对比评测
Comparative Evaluation of LLM-Based Review Paradigms for Academic Paper Quality Assessment
AI 导读
研究在 232 篇分层抽样的 ICLR 2025 投稿上对比五种 LLM 评审范式、三种输入表示和三个模型，发现直接绝对评分与比较排序在和人工评分的秩相关上相当，但会把多数预测压在录用阈值之上，几乎不标记审稿人评为不达标的论文。
正文 · 原文
Abstract:Growing submission volumes have renewed interest in using large language models (LLMs) to support peer review. It remains unclear how such evaluation should be structured, how much the model matters, and whether LLM scores predict research impact. We evaluate five paradigms across three input representations and three LLMs on 232 stratified ICLR 2025 submissions. Direct absolute rating matches comparative ranking on rank correlation with human scores, yet compresses most predictions above the accept threshold, flagging almost none of the papers reviewers scored below it. A base-rate-matched recalibration control locates the cause: comparative ranking supplies placement, not better ordering, and after recalibration Direct Rating reaches higher agreement than Elo while making 24% as many API calls. Abstract-only input beats full-paper summarization, and model choice matters as much as paradigm choice. Replicating on all 11,513 scored submissions reproduces these patterns, shows absolute agreement depends on the evaluated set's score spread, and finds none beating an always-reject baseline in accuracy. Human scores track citations better than any paradigm, with one exception: Group Rating retains its citation correlation after controlling for human scores, and alone ranks within the accepted pool. Ranking quality alone cannot assess review quality; calibration, thresholds, model, and target outcome all matter.
| Comments: | commit to EACL 2027 |
| Subjects: | Computation and Language (cs.CL); Digital Libraries (cs.DL) |
| Cite as: | arXiv:2610.41391 [cs.CL] |
| (or arXiv:2610.41391v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41391 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Fan Huang [view email]
[v1]
Fri, 30 Oct 2026 08:02:25 UTC (605 KB)
来源：arXiv cs.CL · arxiv.org