arXiv cs.CL· Soroush Ebadian, Parand A. Alamdari, Joe Benton, Hailey Joren·· 12 小时前AI 评分45
Tampered ML：评估 LLM 对机器学习研究的判断能力
Tampered ML: Evaluating LLM Judgment of Machine Learning Research
AI 导读
研究者推出 Tampered ML 基准，包含 1,858 组条目对，每组由一篇已发表机器学习论文重构的实验提案与引入特定方法学缺陷的最小改动反事实版本组成，要求模型在两种呈现顺序下识别方法学更严谨的提案。
正文 · 原文
Abstract:One approach to accelerating progress on AI alignment is to automate parts of alignment research itself. Doing so requires systems that can perform reliably on hard-to-supervise research tasks, including identifying flaws in research proposals and evaluating whether a proposed experiment supports its intended claim. To evaluate this capability, we introduce Tampered ML, a benchmark of 1,858 item-pairs, each consisting of an experiment proposal reconstructed from a published machine learning paper and a minimally edited counterfactual that introduces a targeted methodological flaw. Models must identify the more methodologically sound proposal in both presentation orders. A human validation study on a stratified sample estimates that 86% of items are valid (95% CI: 81-91%). Frontier models perform above chance but remain far from reliable (with Opus 5 achieving 77.9% accuracy). They are highly sensitive to the order of presentation and favor flawed designs that appear more specific, transparent, or conventional.
| Comments: | 28 pages, 10 figures. Accepted at EMNLP 2026 (Main Conference) |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Digital Libraries (cs.DL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.32114 [cs.CL] |
| (or arXiv:2610.32114v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32114 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Soroush Ebadian [view email]
[v1]
Fri, 23 Oct 2026 03:32:38 UTC (2,023 KB)
来源：arXiv cs.CL · arxiv.org