跳到正文
arXiv cs.CL· Mukund Athmaram, Manoj Rao, Liang Ma·· 9 小时前AI 评分31

MoE 路由究竟控制什么:对专家轨迹的因果干预研究

What MoE Routing Controls: Causal Interventions on Expert Trajectories

AI 导读

研究通过在生成时给目标专家的 router logit 加固定偏置,把 token 强制导向指定专家路径,在 Qwen3-30B-A3B 上验证了格式路径能改变模型输出文本,而概念路径虽使下一 token 概率偏向该概念、却几乎不影响生成文本。格式一旦启动会自我强化,概念干预效果集中在中间层,其中 religion 和 money 的引导最强。

正文 · 原文

View PDF

Abstract:In mixture-of-experts (MoE) language models, a token is routed to a sequence of experts across layers. We investigate how forcing tokens onto chosen expert paths influences model behavior. We build a target path from the experts that a concept's or role's tokens most often use, and we add a fixed bias to each target expert's router logit during generation to steer the token toward this expert path. We study two kinds of targets. Format targets concern the form of the output, and concept targets concern its topic. In Qwen3-30B-A3B, we show that steering a format path changes the text the model writes. We also show that steering a concept path shifts the model's next-token probabilities toward the concept, even when the path is built in another language, yet this shift has little effect on the generated text. We investigate why formats reach the text but concepts do not by steering only part of the generation, and find evidence that a format, once started, reinforces itself. We show that the concepts religion and money steer most strongly, and that the effect is concentrated in the middle layers. We attempt to explain why the strength of the effect varies by concept through an analysis of embedding distinctiveness and word frequency.
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as: arXiv:2611.22120 [cs.CL]
  (or arXiv:2611.22120v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.22120

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Mukund Athmaram [view email]
[v1] Tue, 17 Nov 2026 03:02:41 UTC (549 KB)

来源:arXiv cs.CL · arxiv.org