arXiv cs.AI· Eug\`ene Berta, David Holzm\"uller, Francis Bach, Michael I. Jordan·· 2 天前AI 评分31
CalArena：大规模事后校准基准发布
CalArena: A Large-Scale Post-Hoc Calibration Benchmark
AI 导读
CalArena 是一个覆盖近 2000 次实验的事后校准标准化基准，涵盖表格与计算机视觉任务及二分类、多分类和大规模分类场景。该基准统一实现了数十种校准方法，并提出用 proper scoring rules 上的 Post-Hoc Improvement（PHI）替代传统校准误差指标来比较方法效果。研究发现平滑校准函数优于分箱方法，高维场景需专用多分类方法，相关数据、代码与评测工具已开源。
正文 · 原文
Abstract:Reliable probability estimates are critical in many machine learning applications, yet modern classifiers are often poorly calibrated. Post-hoc calibration provides a simple and widely used solution, but the large number of proposed methods, combined with small-scale and inconsistent evaluations, makes it difficult to determine which approaches are truly effective in practice. We introduce a large-scale, standardized benchmark for post-hoc calibration, covering nearly 2000 experiments across tabular and computer vision tasks, including binary, multiclass, and large-scale classification settings. Our benchmark aggregates predictions from a diverse set of classical models, modern deep learning architectures, and foundation models, and provides unified, reproducible implementations of dozens of calibration methods within a common evaluation framework. We argue that Post-Hoc Improvement (PHI) in proper scoring rules offers a principled alternative to traditional calibration error estimators for comparing post-hoc methods, capturing both calibration quality and potential degradation to the model's predictive performance. Using this framework, we conduct the most comprehensive empirical study of post-hoc calibration to date. Our results reveal consistent patterns across domains: smooth calibration functions outperform binning-based approaches, dedicated multiclass methods are essential in high-dimensional settings, and generic machine learning models are not competitive without calibration-specific design. To facilitate future research, we release all data, code, and evaluation tools, providing a plug-and-play benchmark for developing and comparing calibration methods.
| Comments: | Accepted at NeurIPS 2026, Evaluations & Datasets track. Camera-ready version with a new ablation study on using log probabilities rather than logits, and a new multiclass post-hoc calibration method: Platt scaling on the logits (one-versus-rest) |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2605.30188 [cs.LG] |
| (or arXiv:2605.30188v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.30188 arXiv-issued DOI via DataCite |
Submission history
From: Eugène Berta [view email]
[v1]
Thu, 28 May 2026 16:31:36 UTC (888 KB)
[v2]
Mon, 1 Jun 2026 16:20:48 UTC (889 KB)
[v3]
Tue, 27 Oct 2026 13:09:15 UTC (1,054 KB)
来源：arXiv cs.AI · arxiv.org