arXiv cs.AI· Shengjie Wang, Mohsen Fayyaz, Nanyun Peng·· 11 小时前AI 评分48
MoE 模型推理路径研究：读取并重塑测试时计算
Routes of Reasoning: Reading and Reshaping Test-Time Compute in MoE Models
AI 导读
研究者发现 MoE 路由器可作为无需训练的推理行为控制接口，通过线性探针在 router logits 上预测六类推理行为标签，效果接近残差流探针且特征量约减半。在推理时选择性编辑专家路由分数，四个模型、四个基准上生成 token 数中位减少 30.4%（最高 63.3%），准确率下降不超过 0.5 个百分点且常有所提升。该信号在覆盖数学、代码、科学、知识、常识五个领域的六个数据集上可泛化。
正文 · 原文
Abstract:Reasoning models allocate test-time compute across distinct behaviors such as planning, deriving, and verifying. However, models frequently "overthink," continuing to verify and re-derive long after finding a correct answer. This wastes expensive compute and can even degrade performance. Existing interventions, such as token budgets or steering, primarily regulate reasoning duration rather than targeting specific behaviors. In this work, we demonstrate that Mixture-of-Experts (MoE) routers provide a natural, training-free interface for behavioral control. We first split reasoning traces into sentence-level segments and label each with one of six reasoning behaviors (planning, execution, restatement, fact, reflection, and conclusion), so that every token carries the behavior of its segment. Linear probes on router logits predict this label nearly as well as probes on the residual stream, the model's full hidden state and the standard target for probing, while using about half as many features. These signals generalize: experts associated with a reasoning behavior on one dataset predict the same behavior on the others, across six datasets spanning five domains (math, code, science, knowledge, and commonsense). Building on this, we selectively edit expert router scores during inference to curb overthinking. Across four models and four benchmarks, this intervention reduces the number of generated tokens by a median of 30.4% (and up to 63.3%). Crucially, it encourages the model to stop once the problem is solved, rather than skipping essential reasoning steps. Accuracy never drops by more than 0.5 percentage points, and frequently improves. Ultimately, MoE routing offers both an interpretable lens into reasoning states and a lightweight mechanism for optimizing test-time compute.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.05919 [cs.AI] |
| (or arXiv:2611.05919v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05919 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shengjie Wang [view email]
[v1]
Wed, 4 Nov 2026 04:13:02 UTC (317 KB)
来源：arXiv cs.AI · arxiv.org