arXiv cs.AI· Ruth Wan Theng Chew, Zhiliang Chen, Apivich Hemachandra, Bryan Kian Hsiang Low·· 13 小时前AI 评分31
BoLT：面向昂贵 LLM 任务的黑盒优化基准发布
BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks
AI 导读
研究者发布 BoLT，首个以 LLM 为中心的黑盒优化基准，通过拟合数千次真实 LLM 实验的轻量代理模型，让 BBO 社区无需承担高昂训练推理成本即可复现实验。BoLT 覆盖多保真度、多目标、异方差噪声和高维搜索空间等 LLM 优化问题，基准测试显示部分 BO 方法在各任务上持续优于其他方法，同时暴露了现有 BBO 方法在 LLM 任务上的不足。
正文 · 原文
Abstract:Optimization of LLM training and inference configurations, such as hyperparameters, data mixtures, and prompts, is critical to performance, but it is often approached heuristically in practice, leading to potentially suboptimal outcomes. By framing them as noisy, expensive, and derivative-free optimization problems, Bayesian optimization (BO) and other black-box optimization (BBO) methods offer a promising yet underexplored direction for principled, sample-efficient methods. However, LLM training and inference costs are prohibitively high for most of the BBO research community, and new methods are often only evaluated on synthetic test functions and small-scale datasets that fail to capture the challenges of modern LLM optimization problems. This impedes the development of BBO methods and makes it difficult to assess their effectiveness on modern LLM tasks. We introduce BoLT, the first LLM-centric benchmark that democratizes LLM research for the BBO community. BoLT is released at this https URL. BoLT covers broad and well-motivated LLM optimization problems, involving multi-fidelity, multi-objective, heteroscedastic noise, and high-dimensional search spaces. Each problem in BoLT is grounded in real experimental data and made fully reproducible and accessible through lightweight surrogate models fitted to the results of thousands of real LLM experiments. We benchmark BoLT against an extensive range of BO and BBO methods, showing that selected BO methods consistently outperform others across tasks and highlighting gaps in existing BBO methods on LLM tasks, underscoring the need to modernize benchmarks for the BBO community.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.17000 [cs.LG] |
| (or arXiv:2605.17000v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.17000 arXiv-issued DOI via DataCite |
Submission history
From: Zhiliang Chen [view email]
[v1]
Sat, 16 May 2026 13:53:44 UTC (2,089 KB)
[v2]
Tue, 27 Oct 2026 03:52:46 UTC (3,510 KB)
来源：arXiv cs.AI · arxiv.org