arXiv cs.AI· Johannes Hog, Neeratyoy Mallik, Steven Adriaensen, Jan Hendrik Metzen, Tom Viering, Danny Stoll, Frank Hutter·· 10 小时前AI 评分34
面向表格基础模型的可靠缩放定律研究
Towards Reliable Scaling Laws for Tabular Foundation Models
AI 导读
研究检验了 TabPFNv2 类表格基础模型的缩放定律能否可靠外推到更大算力预算并泛化到不同评测设置。分析发现合成数据与真实数据上的缩放行为存在差异,拟合良好的缩放定律未必能在未见数据集上给出可靠配置建议,评测数据集过少会系统性高估性能。
正文 · 原文
Abstract:Scaling laws are widely used to predict foundation model performance and guide compute allocation. However, tabular foundation models (TFMs) differ from conventional foundation models in that they are pretrained on virtually unlimited synthetic data from configurable distributions, while ultimately being evaluated on real-world datasets. We investigate whether scaling laws for TabPFNv2-style models reliably extrapolate to larger compute budgets and generalize across evaluation settings. Our analysis reveals differences between synthetic and real-world scaling behavior and shows that accurately fitting a scaling law does not necessarily yield reliable prescriptions on unseen datasets. In particular, selecting configurations using too few evaluation datasets leads to systematically over-optimistic performance estimates. We further demonstrate that compute-optimal scaling requires expanding the range of synthetic pretraining dataset sizes alongside model size and training compute. Increasing the maximum numbers of rows and features sampled during pretraining improves performance and produces more predictable scaling trends. These insights guide the training of more efficient TabPFNv2-like models, using the open-source training pipeline of nanoTabPFN.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.24214 [cs.LG] |
| (or arXiv:2611.24214v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24214 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Johannes Hog [view email]
[v1]
Wed, 18 Nov 2026 10:52:48 UTC (499 KB)
来源:arXiv cs.AI · arxiv.org