跳到正文
arXiv cs.AI· Nicol\'as Astorga, Nabeel Seedat, Mihaela van der Schaar·· 4 小时前AI 评分34

LLM 在扩展搜索空间中的类优化推理:OPT* 任务族与训练方法

Optimization-like Reasoning in LLMs over Expanding Search Spaces

AI 导读

研究者提出 OPT*,一族可扩展的优化式任务,用于训练和评估 LLM 的逐步类优化推理,通过复杂度参数扩大搜索空间且无需新人工标注。方法涵盖以求解器作为价值预言机的在线策略优化,以及无求解器时的基于搜索离线 RL。实验表明在 OPT* 上训练可提升 LLM 的逐步类优化推理能力。

正文 · 原文

View PDF HTML (experimental)

Abstract:Verifiable reward training has improved mathematical and coding reasoning, but these domains capture only part of step-by-step decision making. Many real-world tasks require finding a high-value feasible plan among many valid alternatives. We introduce OPT*, a scalable family of optimization-style tasks for training and evaluating LLM step-by-step optimization-like reasoning along a complexity axis: each task provides a feasibility checker and evaluator, while a complexity parameter expands the search space without requiring new human labels. This motivates studying these tasks in two regimes: (i) solver-guided online policy optimization, which uses a solver as a value oracle for partial states and applies rank-based reward shaping to reinforce better next steps, and (ii) search-based offline RL when such solvers are unavailable. Theoretically, we relate success in large search spaces to the information a reasoner extracts per unit of search budget. Empirically, we ablate the ingredients that make search efficient on OPT* and show that training on OPT* improves step-by-step optimization-like reasoning.
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2606.05464 [cs.AI]
  (or arXiv:2606.05464v2 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2606.05464

arXiv-issued DOI via DataCite

Submission history

From: Nicolas Astorga [view email]
[v1] Wed, 3 Jun 2026 21:43:38 UTC (1,089 KB)
[v2] Tue, 17 Nov 2026 00:52:17 UTC (1,862 KB)

来源:arXiv cs.AI · arxiv.org