arXiv cs.CL· Noy Sternlicht, Simra Shahid, Peter Jansen, Daniel S. Weld, Pao Siangliulue, Tom Hope·· 3 天前AI 评分40
基于 LLM 的新颖性评估为何不稳定：一项系统受控研究
Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation
AI 导读
一项系统受控研究发现，基于 LLM 的新颖性评估表现不佳：仅告知评审者认为某个想法新颖，就能让同一对想法中超过一半的判定发生翻转，成对准确率波动超过 50 个百分点，有时甚至低于随机水平。检索与更大的推理预算帮助有限，两个专门构建的新颖性评估器还不如最便宜的提示词基线。
正文 · 原文
Abstract:AI-based ideation systems are often evaluated on the novelty of their ideas, and that judgment is increasingly delegated to large language models. Such judges are typically built ad hoc and validated, if at all, on human-authored papers rather than on the generated ideas they are meant to score. So, how do novelty judges perform? Not well. We present a systematic controlled study of novelty evaluation design choices. We build an evaluation set automatically, mining OpenReview for passages where reviewers explicitly affirm or dispute a paper's originality and keeping only submissions with unanimous agreement at the extremes of their research area; we pair these with ideas from a vanilla LLM generator. Across six judges, we find that seemingly small prompt design choices have large consequences; e.g., simply telling the judge that reviewers found one idea novel and the other not can change its verdict on more than half of the identical idea pairs it is shown, shifting pairwise accuracy by over 50 points and occasionally pushing it below chance. The same change helps one judge and hurts another. Retrieval and larger reasoning budgets help little, and two purpose-built novelty evaluators are outperformed by our cheapest prompted baseline. These results raise questions about reported novelty gains of automated ideation systems, and call for robust novelty evaluation methods.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.02022 [cs.CL] |
| (or arXiv:2610.02022v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02022 arXiv-issued DOI via DataCite |
Submission history
From: Noy Sternlicht [view email]
[v1]
Thu, 1 Oct 2026 16:43:18 UTC (1,095 KB)
[v2]
Wed, 14 Oct 2026 17:45:03 UTC (1,034 KB)
来源：arXiv cs.CL · arxiv.org