arXiv cs.CL· Shaoliang Yang, Jun Wang·· 11 小时前AI 评分24
论文发现：论文接收预测器排名一致，不代表段落改写评分一致
Ranking Agreement Does Not Reveal Paragraph-Edit Agreement in Two Released Paper Acceptance Predictors
AI 导读
研究比较了一个词与文档特征分类器和两个基于同一 30 亿参数语言模型微调的纯文本接收预测器，在 198 篇 ICLR 论文上考察原始论文与段落改写后的排名一致性。分类器与其中一个预测器排名高度接近、与另一个中等接近，但在"哪些改写能提高分数"上均无可检测的一致性。以完整 32 位精度重打分后编辑一致性仍接近零，每个预测器在五选一中平均原始分数仅比随机选择高不到 2 分（满分 100）。
正文 · 原文
Abstract:A fast acceptance classifier could select revisions for a slower evaluator. We ask whether agreement on existing papers supports that use. We compare a classifier of word and document features with two released text-only acceptance predictors fine-tuned from the same language model with three billion parameters. For 198 papers from the International Conference on Learning Representations, we compare model rankings of the original papers and of paragraph rewrites screened automatically for unchanged facts, claim strength, and attribution. The classifier ranks original papers closely with one predictor and moderately with the other, but shows no detectable agreement with either on which rewrites raise scores. The tests specified in advance used reduced numerical precision. Rounding error dominated the small score responses. We therefore rescored the released weights in full 32-bit precision as a post hoc check; edit agreement remained near zero. At this precision, when each predictor chooses among five rewrites per paper, its average raw score is under two points out of 100 higher than with random choice. The evidence covers presentation rewrites and two open predictors from one family; transfer to costly evaluators remains untested. Revision selectors should therefore be checked against the intended evaluator on the revisions they rank.
| Comments: | 38 pages including supplementary methods and results; 10 figures and 22 tables |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.06789 [cs.CL] |
| (or arXiv:2611.06789v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06789 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shaoliang Yang [view email]
[v1]
Wed, 4 Nov 2026 17:03:30 UTC (607 KB)
来源：arXiv cs.CL · arxiv.org