arXiv cs.CL· Srinivasan Iyengar·· 12 小时前AI 评分38
SPLIT：在不完美验证器下如何分配推理时计算——采样、混合还是验证？
Sample, Mix, or Verify? Allocating Test-Time Compute under Imperfect Verifiers
AI 导读
研究者提出 SPLIT，用少量标注试点数据规划推理时预算，决定采样数量、模型选择与验证开销，其核心是给出 accept/reject 与 top-score 选择下提交正确答案概率的精确公式，并考虑任务难度差异、模型共享盲区与验证器调用相关性。
正文 · 原文
Abstract:A common way to make language models and agents more accurate is to spend more compute at inference time: generate many candidate solutions and submit the one a verifier prefers. Success then requires that some candidate is correct and that the verifier picks it. The first grows predictably with the number of samples; the second does not, because real verifiers (unit tests, majority votes, LLM judges, reward models) accept wrong answers, reject right ones, and make correlated mistakes. With a fixed budget, a practitioner must decide how many samples to draw, which models to draw them from, and how much to spend on verification, and today these choices are made by rules of thumb. We introduce SPLIT, which makes them from a small labelled pilot. It rests on an exact formula for the probability of submitting a correct answer under accept/reject and top-score selection that accounts for varying task difficulty, blind spots shared across models, and correlated repeated verifier calls. SPLIT fits the formula by empirical Bayes and uses it to plan budgets on unseen tasks. Across seven benchmarks spanning math, science QA, code and software-engineering agents, 26 models, 175 agent systems and six kinds of real verifiers, all from public releases, SPLIT lands within 0.02-0.8 percentage points of the best allocation in hindsight, while every fixed rule we test loses at least 2.5 points in some setting. Regret depends only on how well SPLIT ranks configurations, and it stays near-optimal under compute, price and latency costs and online. The formula also explains when asking more LLM judges lowers precision. As a forecaster far beyond the pilot, SPLIT beats power-law fitting when the verifier caps success, and we report where it loses.
| Comments: | 27 pages (18 main + appendix), 14 figures, 11 tables. Code will be released |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| ACM classes: | I.2.6; I.2.7; G.3 |
| Cite as: | arXiv:2610.23255 [cs.LG] |
| (or arXiv:2610.23255v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.23255 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Srinivasan Iyengar [view email]
[v1]
Fri, 16 Oct 2026 10:28:07 UTC (577 KB)
来源：arXiv cs.CL · arxiv.org