arXiv cs.AI· Guy Zyskind·· 10 小时前AI 评分58
研究发现语言模型会把新研究问题说得像已被研究过
Language Models Make New Research Sound Old
AI 导读
arXiv 论文发现,LLM 在验证研究问题时会把开放问题描述成部分已被研究,并夸大相关论文的结论。在 40 个取自新录用论文的问题上,GPT-6-Astra 有 98% 能正确判断问题未被研究,但只有 8% 的普通回答明确说明这一点,六个前沿系统的报告差距达 30-90 个百分点。作者用模型判断加对引用论文的单独核查构建固定答案,把 Astra 清晰报告开放状态的比例从 8% 提升到 95%。
正文 · 原文
Abstract:Researchers are increasingly using large language models (LLMs) to validate research questions. We find that LLMs make open research questions sound partly studied and overstate what related papers establish. Models recognize most of these overstatements when asked to check the papers' full texts separately. On 40 questions drawn from newly accepted papers, GPT-6-Astra correctly judges 98% to be unstudied, but states that status plainly in only 8% of ordinary answers. All six frontier systems show reporting gaps of 30-90 percentage points. Across 22 models given identical papers, higher capability is associated with more accurate judgments and larger reporting gaps. This reporting failure is asymmetric: models correctly identify studied questions in 88-100% of ordinary answers when we supply the answering papers. To correct this, we build a fixed answer from each model's judgment and separate checks of cited papers. This raises Astra's clear reporting of open status from 8% to 95%, with improvements in all six systems. Finally, we predict reviewers' novelty grades by combining several models' written ratings with a rule learned from graded examples, reaching the performance range reported by Think-Probe-Respond. Unlike TPR, our approach needs no access to model weights or internal activations, so it also works with closed-weight frontier models.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23546 [cs.CL] |
| (or arXiv:2611.23546v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23546 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Guy Zyskind [view email]
[v1]
Tue, 17 Nov 2026 21:53:16 UTC (358 KB)
来源:arXiv cs.AI · arxiv.org