arXiv cs.AI· Zhiliang Chen, Alfred Wei Lun Leong, Shao Yong Ong, Apivich Hemachandra, Gregory Kang Ruey Lau, Chuan-Sheng Foo, Zhengyuan Liu, Nancy F. Chen, Bryan Kian Hsiang Low·· 14 小时前AI 评分26
JoBS：用缩放定律预测器重构贝叶斯优化，联合优化 LLM 训练配置
Rethinking Bayesian Optimization for Co-Optimizing LLM Training Configurations
AI 导读
论文提出 JoBS，一种基于贝叶斯优化的方法，可在不每次迭代都完整训练 LLM 的前提下联合优化数据与模型配置。JoBS 先用部分预算学习一个受缩放定律启发的性能预测器，仅凭少量训练步数估计完整训练后的表现，再用剩余预算运行 BO，从而在同等预算内探索更多配置。实验显示其在多种下游任务上优于独立优化数据与模型的方法及现有多保真度 BO 基线，代码已开源。
正文 · 原文
Abstract:Fine-tuning an LLM to maximize performance on a downstream task requires finding the optimal data and model configuration. This is a black-box optimization problem that Bayesian optimization (BO) addresses by sequentially evaluating training configurations and using observed performance feedback to adaptively guide the search towards better configurations. However, directly applying BO to the joint data-model space suffers from two fundamental challenges: the prohibitive cost of full LLM training at each iteration, and the large number of BO iterations required for effective exploration due to the high problem dimensionality. This paper introduces JoBS, a BO-based approach that co-optimizes data and model configurations without requiring a full training run at every iteration. JoBS allocates an initial portion of the optimization budget to learn a scaling-law-inspired performance predictor that estimates fully trained LLM performance from only a small number of training steps. The remaining budget is then used to run BO with this predictor, eliminating the need for full training runs and enabling JoBS to explore significantly more data and model configurations within the same budget. We analyze JoBS's average regret and derive the optimal budget allocation between predictor learning and BO iterations. Empirically, JoBS outperforms independent data and model optimization methods and existing multi-fidelity BO baselines across a diverse set of downstream tasks. Our code is available at this https URL.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2602.08351 [cs.LG] |
| (or arXiv:2602.08351v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.08351 arXiv-issued DOI via DataCite |
Submission history
From: Zhiliang Chen [view email]
[v1]
Mon, 9 Feb 2026 07:33:40 UTC (7,986 KB)
[v2]
Thu, 19 Feb 2026 01:32:53 UTC (7,986 KB)
[v3]
Wed, 28 Oct 2026 07:55:31 UTC (7,655 KB)
来源：arXiv cs.AI · arxiv.org