arXiv cs.AI· Binh Duc Vu, David S. Watson·· 10 小时前AI 评分31
树集成的极小极大收敛率与谱蒸馏
Minimax Rates and Spectral Distillation for Tree Ensembles
AI 导读
研究从谱视角分析随机森林（RF）与梯度提升机（GBM），推导出 RF 回归的极小极大最优收敛率，并证明在树生长的温和正则条件下，诱导核算子的特征值衰减决定统计收敛速度。基于该谱视角提出压缩方案：RF 用核算子的前导特征函数、GBM 用平滑矩阵的前导奇异向量学习非线性映射，蒸馏模型比原模型小几个数量级且保持有竞争力的预测性能。方法在森林剪枝与规则提取上优于现有算法，面向资源受限计算场景。
正文 · 原文
Abstract:Tree ensembles such as random forests (RFs) and gradient boosting machines (GBMs) are among the most widely used supervised learners, yet their theoretical properties remain incompletely understood. We adopt a spectral perspective on these algorithms, with two main contributions. First, we derive minimax-optimal convergence for RF regression, showing that, under mild regularity conditions on tree growth, the eigenvalue decay of the induced kernel operator governs the statistical rate. Second, we exploit this spectral viewpoint to develop compression schemes for tree ensembles. For RFs, leading eigenfunctions of the kernel operator capture the dominant predictive directions; for GBMs, leading singular vectors of the smoother matrix play an analogous role. Learning nonlinear maps for these spectral representations yields distilled models that are orders of magnitude smaller than the originals while maintaining competitive predictive performance. Our methods compare favorably to state of the art algorithms for forest pruning and rule extraction, with applications to resource constrained computing.
| Comments: | NeurIPS camera ready version. 10 pages main text, 48 pages total (excluding checklist), with 12 figures and 10 tables total |
| Subjects: | Machine Learning (stat.ML); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.11841 [stat.ML] |
| (or arXiv:2605.11841v3 [stat.ML] for this version) | |
| https://doi.org/10.48550/arXiv.2605.11841 arXiv-issued DOI via DataCite |
Submission history
From: Binh Vu [view email]
[v1]
Tue, 12 May 2026 09:26:00 UTC (2,627 KB)
[v2]
Sun, 8 Nov 2026 20:04:41 UTC (2,634 KB)
[v3]
Tue, 10 Nov 2026 11:02:27 UTC (2,634 KB)
来源：arXiv cs.AI · arxiv.org