arXiv cs.CL· Guanzhi Deng, Zengyan Liu, Haibo Wang, Kuan Wu, Shing Yin Wong, Sichun Luo, Zhuoran Wang, Linqi Song·· 13 小时前AI 评分34
强 MoE 模型即可实现过程监督：利用专家路由构建 PRM
A Strong MoE Is All You Need for Process Supervision
AI 导读
研究提出利用 MoE 模型的专家路由统计作为内在过程置信信号，其效果可与传统输出 logit 置信度竞争。基于此构建的 RoCo-Rethink 能定位最不确定的推理步骤并选择性重新生成后续轨迹，RoCo-PRM 则用强 MoE 自动标注步骤分数并蒸馏为独立 PRM。在多种 MoE 架构和推理任务上，两者均持续优于强基线，且 RoCo-PRM 可跨模型架构与任务迁移。
正文 · 原文
Abstract:Process reward models (PRMs) provide fine-grained supervision for multi-step reasoning, yet obtaining reliable process labels remains a central bottleneck. Existing approaches typically rely on human annotation, rollout-based estimation, or auxiliary verification. In this work, we explore a different source of process supervision natively exposed by Mixture-of-Experts (MoE) models: expert routing. We show that simple routing statistics provide an intrinsic process-confidence signal that is competitive with conventional output-logit confidence. Building on this observation, we construct routing confidence by combining overall routing mass, head routing mass, and within-step routing variation, and introduce RoCo-Rethink, which identifies the least confident reasoning step and selectively regenerates the subsequent trajectory. We further introduce RoCo-PRM, which uses a strong MoE to automatically annotate reasoning trajectories with routing-derived step scores and distills them into a standalone PRM. Across multiple MoE architectures and diverse reasoning tasks, both RoCo-Rethink and RoCo-PRM consistently outperform strong baselines, while RoCo-PRM transfers effectively across both model architectures and tasks. Our findings suggest that expert routing is more than a computational dispatch mechanism: it provides an architecture-native source of process supervision for reasoning.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.29498 [cs.CL] |
| (or arXiv:2610.29498v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29498 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Guanzhi Deng Mr. [view email]
[v1]
Wed, 21 Oct 2026 09:51:01 UTC (406 KB)
来源：arXiv cs.CL · arxiv.org