arXiv cs.CL· Garvin Kruthof·· 11 小时前AI 评分47
DriftBench：多轮 LLM 科学构思中的约束遵循基准
Models Recall What They Violate: Constraint Adherence in Multi-Turn LLM Ideation
AI 导读
研究者提出 DriftBench 基准，用 24 个科学领域的 38 份研究简报、7 个模型（来自 5 家提供商，含 2 个开源权重模型）和 4 种交互条件共 2116 次运行，评测多轮 LLM 辅助科学构思中的约束遵循。
正文 · 原文
Abstract:When researchers refine ideas with large language models over several turns, do the models preserve the original brief's constraints? We introduce DriftBench, a benchmark for constraint adherence in multi-turn LLM-assisted scientific ideation. It pairs 38 research briefs from 24 scientific domains, each with hard constraints and banned moves, with four interaction conditions and seven models from five providers (two open-weight), scored on 2,116 runs. Iterative pressure to improve a proposal raises structural complexity in every model and lowers adherence to the brief's hard constraints. A restatement probe shows that the constraints stay legible and reproducible on demand while the proposal departs from them. The knows-but-violates (KBV) rate, the share of pressure runs scored non-compliant on a hard constraint despite a correct restatement, exceeds 50% for most models. A crossed-judge design measures the judge offset. GPT-5.4 scores adherence 0.535 points lower than Opus 4.6 on the same runs. The design finds no judge-by-condition interaction on adherence, so the pressure effect on adherence is judge-stable while KBV levels and the bottom of the model ranking are not. Re-eliciting the brief at checkpoints lowers KBV for some models but not others, and complexity inflation persists. Blind human raters find more constraint violations than the LLM judge, so reported adherence is conservative. We release all briefs, prompts, rubrics, transcripts, and scores.
| Comments: | NeurIPS 2026 Evaluations and Datasets Track, 41 pages, 6 figures. Code: this https URL Data: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2604.28031 [cs.CL] |
| (or arXiv:2604.28031v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.28031 arXiv-issued DOI via DataCite |
Submission history
From: Garvin Kruthof [view email]
[v1]
Thu, 30 Apr 2026 15:46:33 UTC (58 KB)
[v2]
Mon, 4 May 2026 01:39:21 UTC (59 KB)
[v3]
Fri, 6 Nov 2026 16:12:20 UTC (240 KB)
来源：arXiv cs.CL · arxiv.org