FrontierOR：评测 LLM 在大规模优化问题中设计高效算法的能力
FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization
FrontierOR 是首个基于文献构建的基准，用于评测 LLM 在真实大规模优化问题上生成高效算法的能力，包含 180 个源自顶级运筹学期刊论文的任务及隐藏的专家验证评测集。结果显示，最强 LLM 在 Full-set 任务实例上仅成功 39%，成功定义为在 Gurobi 运行时间内返回可行解且目标差距不超过 1%；在 Hard 子集上，测试时进化将最强单次模型的比例从 51% 提升至 78%。
Authors:Minwei Kong, Chonghe Jiang, Ao Qu, Wenbin Ouyang, Zhaoming Zeng, Xiaotong Guo, Zhekai Li, Junyi Li, Yi Fan, Xinshou Zheng, Xi Jing, Yikai Zhang, Zhiwei Liang, Seonghoo Kim, Runqing Yang, Zijian Zhou, Sirui Li, Han Zheng, Wangyang Ying, Ou Zheng, Chonghuan Wang, Jinglong Zhao, Hanzhang Qin, Cathy Wu, Paul Pu Liang, Jinhua Zhao, Hai Wang
Abstract:Large language models (LLMs) are increasingly used for optimization modeling and solver-code generation, yet practical operations research and optimization problems often require a harder capability: designing scalable algorithms that exploit problem structure and outperform direct formulation-and-solve baselines. Existing benchmarks are limited to small or simplified examples far below real-world scale and complexity. We introduce FrontierOR, the first literature-grounded benchmark targeting LLM-generated algorithm efficiency on realistic large-scale optimization problems. FrontierOR includes 180 diverse tasks, each derived from a paper published in a top-tier operations research venue, with standardized instances and a hidden, expert-verified evaluation suite. We evaluate recent LLMs' one-shot algorithm generation and improvements achieved through several self-evolving frameworks. The results show that current models still struggle to move from executable formulations to efficient optimization algorithms. Even the strongest LLM succeeds on only 39% of Full-set task instances, where success is defined as returning a feasible solution with an objective gap of at most 1% relative to Gurobi within its runtime. On the Hard subset, test-time evolution raises this proportion from 51% for the strongest one-shot model to 78%. FrontierOR provides a continuously maintained testbed for evaluating LLM-designed optimization algorithms under a shared protocol and controlled computational settings. Code and data are publicly released at this https URL.
| Comments: | Accepted as a poster at NeurIPS 2026 (Evaluations and Datasets Track) |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.25246 [cs.AI] |
| (or arXiv:2605.25246v4 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.25246 arXiv-issued DOI via DataCite |
Submission history
From: Minwei Kong [view email]
[v1]
Sun, 24 May 2026 20:10:42 UTC (1,785 KB)
[v2]
Tue, 26 May 2026 13:18:52 UTC (1,785 KB)
[v3]
Sat, 30 May 2026 22:51:15 UTC (1,785 KB)
[v4]
Wed, 11 Nov 2026 22:12:04 UTC (1,654 KB)
来源：arXiv cs.AI · arxiv.org