跳到正文
arXiv cs.AI· Mridul Mahajan, Xuezhou Zhang, Aldo Pacchiano·· 23 小时前AI 评分31

分层策略:将推理时采样预算分配给多个 LoRA 策略以提升 pass@k

Learning Stratified Policies for Inference-Time Objectives

AI 导读

研究提出“分层策略”:在共享语言模型主干上挂载 k 个独立 LoRA 适配器,每个策略贡献一个回复,共同构成 k 样本推理时目标,并针对 pass@k 优化。理论分析显示,在单个 LoRA 预测误差严格为正的线性设定下,LoRA 群体可实现零 best-of-k 近似误差。多个推理任务上,该方法对 pass@k 的改善稳定优于单策略基线。

正文 · 原文

View PDF HTML (experimental)

Abstract:Inference-time scaling improves the performance of large language models by sampling multiple responses and evaluating their collective utility. However, many existing approaches obtain these samples by repeatedly sampling from a single policy. In this work, we study if the sample budget at inference time can instead be allocated across a population of policies. We introduce stratified policies, where each of $k$ learned policies contributes one response to a $k$-sample inference-time objective. We realize this policy population using separate LoRA adapters on a shared language model backbone and optimize it for pass@$k$. We theoretically motivate the approach by analyzing the linear setting; a population of LoRAs can achieve zero best-of-$k$ approximation error in settings where every single LoRA has strictly positive prediction error. Empirically, we demonstrate that stratified policies consistently improve pass@$k$ over a single-policy baseline across multiple reasoning tasks. Overall, our results highlight how distributing samples across complementary policies can be a more effective use of sample budget than repeated i.i.d. sampling from a single policy at inference time.
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.23154 [cs.LG]
  (or arXiv:2611.23154v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2611.23154

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Mridul Mahajan [view email]
[v1] Tue, 17 Nov 2026 18:06:12 UTC (1,218 KB)

来源:arXiv cs.AI · arxiv.org