跳到正文
arXiv cs.AI· Dario Deniz Ergun, Haohua Wang, Brendon Tao, Jonah Kollenberg, Chen Ye Gan·· 10 小时前AI 评分31

稀疏 AI 评测中不确定性量化的快速贝叶斯方法

Fast Bayesian Methods for Uncertainty Quantification in Sparse AI Evaluation

AI 导读

一种轻量贝叶斯预测器无需大模型即可估计 AI 系统在新基准上的表现:用公开评测计数选取弱 Beta 先验,再用最多 31 个已揭示结果精确更新。其在 AIMS 竞赛的在线评测中取得团队报告的 Brier ALC 0.188477,较常数 0.5 基线降低 24.6%,完整提交仅 7,754 字节,服务器微基准为每次预测 12.71 微秒。

正文 · 原文

View PDF HTML (experimental)

Abstract:Estimating how an AI system will perform on a new benchmark need not require another large model. We use public evaluation counts to choose a weak Beta prior, then update it exactly from at most 31 revealed outcomes. The predictor uses no item semantics, pretrained weights, or inference API. Its single online evaluation in the AIMS competition returned a team-reported Brier ALC of 0.188477, a 24.6% reduction from the constant-0.5 reference. The complete submission is 7,754 bytes; a server microbenchmark measures 12.71 microseconds per prediction with 31 matching labels. We describe how source evaluations are compressed into 301 prior means and show why a concentration of two lets new evidence rapidly determine the prediction. The resulting posterior gives exact uncertainty about the modelled success rate without numerical integration or neural inference. A public evaluation covers 79 system-benchmark pairs and eight seeds, reaching Brier ALC 0.200250 for the submitted method. Accompanying code and sufficient statistics reconstruct the fitted prior byte for byte and replay all 632 evaluation trajectories. The result demonstrates how much predictive value a compact Bayesian model can extract from sparse target feedback at very low online cost.
Comments: 13 pages, 5 figures. Accepted at NeurIPS 2026 AI Measurement Science Workshop
Subjects: Artificial Intelligence (cs.AI); Computation (stat.CO)
ACM classes: I.2.6
Cite as: arXiv:2611.24159 [cs.AI]
  (or arXiv:2611.24159v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.24159

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Chen Ye Gan [view email]
[v1] Wed, 18 Nov 2026 10:02:38 UTC (1,514 KB)

来源:arXiv cs.AI · arxiv.org