arXiv cs.AI· Jonas Landsgesell, Pascal Knoll, Tizian Wenzel·· 3 天前AI 评分22
ScoringBench：用恰当评分规则评估表格基础模型的基准
ScoringBench: A Benchmark for Evaluating Tabular Foundation Models with Proper Scoring Rules
AI 导读
研究者推出 ScoringBench，一个开放可扩展的基准，用（加权）CRPS、能量分数和区间分数等恰当评分规则评估表格回归模型，覆盖 101 个数据集，并提供基于 git 的排行榜和两种排名协议。对 TabPFN、TabICL、ExaOneTabular、Nori 等表格基础模型、梯度提升树和 MLP 的评估显示，模型排名会随评分规则变化，点估计指标表现好的模型在概率指标上可能排名更低。
正文 · 原文
Abstract:Tabular foundation models such as TabPFN, TabICL, ExaOneTabular, and Nori already produce full predictive distributions, yet prevailing regression benchmarks evaluate them almost exclusively via point-estimate metrics such as RMSE. This overlooks the distributional information these models are designed to provide. We introduce ScoringBench, an open and extensible benchmark that evaluates tabular regression models under a comprehensive suite of proper scoring rules - including (weighted) CRPS, energy scores and interval scores - alongside standard point metrics. ScoringBench covers 101 regression datasets from diverse domains, supports transparent community contributions via a git-based leaderboard, and provides two complementary ranking protocols: an ordinal Demsar/autorank approach and a magnitude-preserving $z$-score ranking approach. Our evaluation of various model types, including (fine-tuned) tabular foundation models, gradient-boosted trees, and MLPs, shows that model rankings shift depending on the chosen scoring rule. Models that perform well on point-estimate metrics can rank lower on probabilistic ones. These findings demonstrate that evaluation metric selection is a core modeling decision and a domain-specific requirement for applications where certain types of errors carry higher consequences.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.29928 [cs.AI] |
| (or arXiv:2603.29928v4 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2603.29928 arXiv-issued DOI via DataCite |
Submission history
From: Jonas Landsgesell [view email]
[v1]
Tue, 31 Mar 2026 16:01:56 UTC (667 KB)
[v2]
Mon, 27 Apr 2026 17:23:20 UTC (1,757 KB)
[v3]
Mon, 4 May 2026 14:43:11 UTC (1,920 KB)
[v4]
Mon, 12 Oct 2026 08:36:09 UTC (2,724 KB)
来源：arXiv cs.AI · arxiv.org