arXiv cs.CL· Shenghao Ye, Yu Guo, Zhengheng Li, Qiang Huang, Shuangwu Chen, Wang Luo, Qian Rui, Jie Zhang, Tao Zhang, Dong Jin, Xiaobin Tan, Quan Zheng, Jian Yang·· 12 小时前AI 评分27
RoRo：用评分标准引导的过程奖励优化逐步模型路由
Rubric-Guided Process Reward for Stepwise Model Routing
AI 导读
针对逐步模型路由中结果奖励无法评估中间路由决策的问题，研究者提出 RoRo 框架，通过构建基于结果、成本与过程质量的路由轨迹偏好对，交替训练 Rubricor 生成查询专属评分标准、Judge 据此打分，得到的过程奖励与结果奖励结合后用 GRPO 优化路由策略。在五个推理基准的同族与跨族设置下，RoRo 均优于强基线，取得更好的准确率与成本权衡。
正文 · 原文
Authors:Shenghao Ye, Yu Guo, Zhengheng Li, Qiang Huang, Shuangwu Chen, Wang Luo, Qian Rui, Jie Zhang, Tao Zhang, Dong Jin, Xiaobin Tan, Quan Zheng, Jian Yang
Abstract:Stepwise model routing improves the efficiency of Large Reasoning Models (LRMs) by assigning each reasoning step to a suitable model. Recent methods formulate routing as a sequential decision process and train the router with reinforcement learning. However, although they model routing as a process, they still supervise the router with outcome rewards. Such rewards only reflect final answer correctness and fail to evaluate intermediate routing decisions, which can weaken performance and generalization. To address this gap, we propose RoRo, a rubric-guided process reward framework for stepwise model routing. RoRo first collects diverse routing trajectories and constructs preference pairs based on outcome, cost, and process quality. It then trains a Rubricor to generate a query-specific evaluation rubric and a Judge to score routing trajectories under this rubric through alternating optimization. The resulting process rewards are combined with outcome rewards to optimize the routing policy via GRPO. Experiments on five reasoning benchmarks under both same-family and cross-family settings show that RoRo consistently outperforms strong baselines and achieves better accuracy and cost trade-offs.
| Comments: | 17 pages, 9 figures, accepted to EMNLP 2026 Main |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.29310 [cs.AI] |
| (or arXiv:2605.29310v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.29310 arXiv-issued DOI via DataCite |
Submission history
From: Ye Shenghao [view email]
[v1]
Thu, 28 May 2026 03:42:24 UTC (465 KB)
[v2]
Sat, 19 Sep 2026 15:25:58 UTC (475 KB)
[v3]
Sun, 1 Nov 2026 02:30:45 UTC (475 KB)
来源：arXiv cs.CL · arxiv.org