arXiv cs.AI· Mridul Mahajan, Xuezhou Zhang, Aldo Pacchiano·· 23 小时前AI 评分31
分层策略:将推理时采样预算分配给多个 LoRA 策略以提升 pass@k
Learning Stratified Policies for Inference-Time Objectives
AI 导读
研究提出“分层策略”:在共享语言模型主干上挂载 k 个独立 LoRA 适配器,每个策略贡献一个回复,共同构成 k 样本推理时目标,并针对 pass@k 优化。理论分析显示,在单个 LoRA 预测误差严格为正的线性设定下,LoRA 群体可实现零 best-of-k 近似误差。多个推理任务上,该方法对 pass@k 的改善稳定优于单策略基线。
正文 · 原文
Abstract:Inference-time scaling improves the performance of large language models by sampling multiple responses and evaluating their collective utility. However, many existing approaches obtain these samples by repeatedly sampling from a single policy. In this work, we study if the sample budget at inference time can instead be allocated across a population of policies. We introduce stratified policies, where each of $k$ learned policies contributes one response to a $k$-sample inference-time objective. We realize this policy population using separate LoRA adapters on a shared language model backbone and optimize it for pass@$k$. We theoretically motivate the approach by analyzing the linear setting; a population of LoRAs can achieve zero best-of-$k$ approximation error in settings where every single LoRA has strictly positive prediction error. Empirically, we demonstrate that stratified policies consistently improve pass@$k$ over a single-policy baseline across multiple reasoning tasks. Overall, our results highlight how distributing samples across complementary policies can be a more effective use of sample budget than repeated i.i.d. sampling from a single policy at inference time.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23154 [cs.LG] |
| (or arXiv:2611.23154v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23154 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mridul Mahajan [view email]
[v1]
Tue, 17 Nov 2026 18:06:12 UTC (1,218 KB)
来源:arXiv cs.AI · arxiv.org