arXiv cs.CL· Hieu Tran, Zonghai Yao, Sunjae Kwon, Hong Yu·· 13 小时前AI 评分31
先度量再分配：面向强化学习中 MCP 智能体的校准 rollout 预算
Measure First, Then Allocate: Calibrated Rollout Budgets for MCP Agents in Reinforcement Learning
AI 导读
研究提出 BRACE（Bank-calibrated Rollout Allocation under Correlated Episodes），在生成前预测混合奖励组，从而为 MCP 环境中的策略训练分配 rollout 预算。
正文 · 原文
Abstract:Training tool-using language models requires costly interaction, yet identical rewards within a rollout group provide no group-relative learning signal. We use reward disagreement to guide harness evaluation and the allocation of rollouts for policy training in Model Context Protocol (MCP) environments. We first introduce Discord, which tests harness edits using exact paired inference and decision-preserving early stopping: only disagreeing paired outcomes favor one harness over another. This insight motivates prospective allocation in BRACE (Bank-calibrated Rollout Allocation under Correlated Episodes). BRACE combines historical task evidence, uncertainty in task success rates, and a correlated reward model to forecast mixed-reward groups before allocating generation budget. Controlled harness experiments favor Discord, while live MCP experiments expose a retention-cost trade-off. At three Qwen3-VL model scales, BRACE exceeds the strongest plotted baseline peaks with fewer new training rollouts; historical-bank collection is counted separately. It also leads the compared methods on BFCL and NESTFUL without further training. Ablations probe the bank, estimator, and allocation score; prospective evaluation checks forecast calibration. These results support forecasting reward contrast before generation.
| Comments: | 21 pages, 7 figures, 6 tables |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.28785 [cs.LG] |
| (or arXiv:2610.28785v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.28785 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hieu Tran [view email]
[v1]
Tue, 20 Oct 2026 18:15:52 UTC (158 KB)
来源：arXiv cs.CL · arxiv.org