arXiv cs.AI· Siyu Zhou, Patrick Vossler, Venkatesh Sivaraman, Yifan Mai, Jean Feng·· 13 小时前AI 评分31
面向 AI 系统的自适应审计：SAVI 框架提供随时有效保证
Adaptive auditing of AI systems with anytime-valid guarantees
AI 导读
研究者提出基于 Safe Anytime-Valid Inference（SAVI）的假设检验框架，将自适应审计形式化为"以赌代测"，同时检验模型无失败模式的零假设与审计者采样策略的零假设。
正文 · 原文
Abstract:A major bottleneck in characterizing the failure modes of generative AI systems is the cost and time of annotation and evaluation. Consequently, adaptive testing paradigms have gained popularity, where one opportunistically decides which cases and how many to annotate based on past results. While this framework is highly practical, its extreme flexibility makes it difficult to draw statistically rigorous conclusions, as it violates classical assumptions: the number of observations is typically limited (often 10 to 50 cases) and decisions regarding sampling and stopping are made in the midst of data collection rather than based a pre-specified rule. To characterize what statistical inferences can be drawn from highly adaptive audits, we introduce a hypothesis testing framework from two "dueling" perspectives: (i) the model's null that asserts there is no failure mode with performance below a target threshold versus (ii) the auditor's null that asserts they have a sampling strategy that will uncover a failure mode. Leveraging Safe Anytime-Valid Inference (SAVI), we formalize the auditor as conducting "testing by betting," which translates into simultaneous e-processes for testing the dueling null hypotheses. Furthermore, if the auditor is sufficiently powerful, we prove that these two hypotheses are asymptotically inverses of each other, in that passage of such an audit certifies the AI system as being globally robust. Empirically, we demonstrate that our proposed testing procedures maintain anytime-valid Type-I error control, outperform pre-specified testing methods, and can reach statistically rigorous conclusions sometimes with as few as 20 observations.
| Comments: | 2026 Conference on Neural Information Processing Systems |
| Subjects: | Artificial Intelligence (cs.AI); Statistics Theory (math.ST); Machine Learning (stat.ML) |
| Cite as: | arXiv:2605.07002 [cs.AI] |
| (or arXiv:2605.07002v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.07002 arXiv-issued DOI via DataCite |
Submission history
From: Jean Feng [view email]
[v1]
Thu, 7 May 2026 22:33:48 UTC (1,044 KB)
[v2]
Wed, 21 Oct 2026 02:55:26 UTC (1,112 KB)
来源：arXiv cs.AI · arxiv.org