arXiv cs.AI· Feilian Huang·· 9 天前AI 评分24
ICLR 2017-2025 同行评审噪声观测估计器：The Review Lottery
The Review Lottery: Benchmarking an Observational Estimator of Peer-Review Noise (ICLR 2017-2025)
AI 导读
研究者构建了一个仅凭公开评审数据即可估计会议审稿噪声的观测估计器，并在 NeurIPS 2021 上完成外部校准：k=3 时模拟分歧率 23.3%，与报告的 23.0% 仅差 0.3 个百分点。
正文 · 原文
Abstract:How much of a conference accept/reject decision would change if the same paper were reviewed by a different set of reviewers? Running a second independent program committee is the gold standard for answering this, but it is prohibitively expensive: done only twice (NeurIPS 2014 and 2021). We build an observational estimator of this quantity from public review data alone, calibrate it twice, and apply it to nine years of ICLR (2017-2025; 36,113 papers, 134,912 reviews). The estimator decomposes scores with a Bayesian ordered-probit model into paper quality and reviewer noise, maps scores to decisions with a logistic model, and simulates two independent committees (posterior draws B=1,000; committee sizes k=2,3,4). Estimated disagreement rates are 23-30% at k=2 and 18-24% at k=4; 30-50% of accepted papers would be rejected. External calibration: at the NeurIPS 2021 reviewer-count caliber (k=3), the simulated 2021 disagreement rate is 23.3% [21.7%, 25.0%] vs. reported 23.0% (bias +0.3pp); accept precision and committee correlation agree within 5pp and 0.04. Internal calibration: on 18,740 papers with 4+ reviews, random model-free 2+2 reviewer splits agree with the k=2 simulation within 1pp in 2018 and 2021-2025. Longitudinally, we find no robust time trend in reviewer noise over 2017-2025. The high accepted-paper flip rates of 2020 and 2021 have distinct mechanisms: the 2020 four-point scale compressed scores (23.7% of papers had zero within-paper variance), and a counterfactual shows coarsening the scale raises disagreement by about 7pp; 2021 instead combined the lowest signal-to-noise ratio in the sample with the most threshold-crowded acceptances. For the LLM era, a 2023 breakpoint test on within-paper score variance finds no break, but the design has almost no power, and no post-2022 review text or confidence data exist, so no LLM attribution is attempted.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.06591 [cs.AI] |
| (or arXiv:2610.06591v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.06591 arXiv-issued DOI via DataCite |
Submission history
From: Feilian Huang [view email]
[v1]
Mon, 5 Oct 2026 16:05:17 UTC (765 KB)
[v2]
Thu, 8 Oct 2026 05:16:30 UTC (969 KB)
来源：arXiv cs.AI · arxiv.org