arXiv cs.AI· Ramnarayan Choudhary·· 12 小时前AI 评分31
自适应实验可能掩盖模型误差：检测限与证伪预算
Adaptive Experimentation Can Hide Model Error:Detection Limits and Falsification Budget
AI 导读
研究发现自适应实验设计会反复选择错误模型恰好拟合的设置，从而掩盖模型本身的错误。作者基于实验者自身的采样分布界定了任何有效性检验的效力上限，并提出一种可计算的诊断指标，用于衡量相对于工作模型声明扩展的未解决误差，进而给出一条无需训练的规则：预留部分设计预算用于证伪。
正文 · 原文
Abstract:An experiment can be informative about the parameters of a working model and uninformative about whether that model is wrong. Adaptive design can reinforce this mismatch by repeatedly selecting settings where an incorrect model fits. We bound the power of any valid adequacy test in terms of the experimenter's own sampling distribution, then distinguish information absent from the selected data from information lost by the detector. A computable diagnostic measures unresolved error relative to a declared expansion of the working model. It motivates a training-free rule that reserves part of the design budget for falsification. The empirical record is mixed: the independently specified primary study fails detection eligibility, and a later numerator diagnostic fails its improvement criteria. After the frozen rule fails on fresh continuous-null panels, a disclosed secondary horizon-calibrated revision detects 35.8% of misspecified campaigns versus 31.7% for a class-separation control at T = 240, but has no resolved advantage at T = 480. In a corrected 289-unit measured replay it improves on random design and trails class-discrimination EIG. A conditional futility bound explains why remaining time matters without proving the gate constant or its power. These results support separating parameter learning, model checking and detector performance; they do not establish a universally superior design policy or prospective scientific discovery.
| Subjects: | Methodology (stat.ME); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.07529 [stat.ME] |
| (or arXiv:2611.07529v1 [stat.ME] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07529 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ramnarayan Choudhary [view email]
[v1]
Thu, 5 Nov 2026 06:40:39 UTC (247 KB)
来源：arXiv cs.AI · arxiv.org