arXiv cs.AI· Guoqiang Zhang, Kexin Tan, Ming Zhang, Li Ju, Wenqing Jing, Zhonghan Yue, Jiayi Chen, Shiqiang Wu, Shaofan Liu, Yue Zhang, Yuankai Ying, Yang Shi, Tao Gui, Qi Zhang, Xuanjing Huang·· 14 小时前AI 评分35
NovGauge：细粒度诊断 LLM 论文新颖性评估能力的基准
NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment
AI 导读
研究者提出 NovGauge，一个面向 LLM 新颖性评估诊断的细粒度基准，包含 619 组论文对和 50 个多论文集合，按任务、问题、方法三个维度独立标注。对 18 个 LLM 的评估显示，各维度幻觉率为 0%-39%，在非幻觉的正确正例判断中超过 70% 的引用证据无法在逻辑上支撑所给理由。
正文 · 原文
Authors:Guoqiang Zhang, Kexin Tan, Ming Zhang, Li Ju, Wenqing Jing, Zhonghan Yue, Jiayi Chen, Shiqiang Wu, Shaofan Liu, Yue Zhang, Yuankai Ying, Yang Shi, Tao Gui, Qi Zhang, Xuanjing Huang
Abstract:Large language models (LLMs) are increasingly used in peer review at major AI conferences, yet novelty remains a persistent weak point. Existing benchmarks assess novelty as a single holistic score, making it difficult to diagnose which dimension a model misjudges or whether its evidence is faithful. We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis. The benchmark contains 619 paper pairs and 50 multi-paper sets, drawn from two expert sources: ICLR reviewer overlap claims and survey co-citations. Instances are independently labeled along three dimensions: task, problem, and method, capturing application goals, technical challenges, and solution approaches. We propose a cascading diagnostic pipeline that verifies per-dimension correctness, evidence grounding, and logical support. Evaluation of 18 LLMs shows hallucination rates ranging from 0% to 39% across dimensions, and among non-hallucinated correct-positive judgments, over 70% cite evidence fails to logically support the stated reason. The best-performing model, GPT-5.5, achieves 43-72% Verified F1 across dimensions, while most models retain less than half of their raw F1 after faithfulness verification. These results suggest that current LLMs remain far from reliable scientific novelty assessment, particularly when correctness is conditioned on faithful evidence grounding.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.11234 [cs.AI] |
| (or arXiv:2609.11234v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.11234 arXiv-issued DOI via DataCite |
Submission history
From: Guoqiang Zhang [view email]
[v1]
Thu, 10 Sep 2026 08:34:56 UTC (701 KB)
[v2]
Thu, 29 Oct 2026 03:51:41 UTC (701 KB)
来源：arXiv cs.AI · arxiv.org