arXiv cs.AI· Yifan Wang, Jinyi Mu, Mayank Jobanputra, Yu Wang, Ji-Ung Lee, Soyoung Oh, Isabel Valera, Vera Demberg·· 14 小时前AI 评分34
稀疏与多样性正则化让 MoE 奖励模型专家更可解释、可组合，少样本个性化提升 25.81 分
Interpretability Is Not Just a Tax: Sparsity and Diversity Yield Legible and Composable Reward Experts for Personalization
AI 导读
研究审计了 Mixture-of-Experts（MoE）奖励模型的专家结构，发现传统 MoE 学到的专家相互纠缠，问题源于训练目标而非架构本身。加入现成的稀疏与多样性正则化器后，所得稀疏 MoE 在受控设定下专家纯度达 84.8%（传统 MoE 为 60.8%），专家可组合，仅用 50 个样本做轻量 router 更新即带来 25.81 分的少样本个性化提升，约为纠缠专家的 2.7 倍。
正文 · 原文
Abstract:Preference modeling aims to capture heterogeneous human values and increasingly relies on learning latent components to represent this variation without fine-grained supervision. What these components actually represent, however, is rarely examined. We audit Mixture-of-Experts (MoE) reward models along three rungs---semantic coherence, functional correspondence, and composability---in controlled and real-world settings, holding the architecture and training data fixed so that differences isolate the effect of the training objective. Our diagnosis is that the components conventional MoE reward models learn are entangled, which limits not only interpretability but also the personalization such models are built for. The entanglement is not intrinsic to the architecture but a property of the training objective: off-the-shelf sparsity and diversity regularizers already substantially reduce it. The resulting configuration, which we call sparse MoE, recovers all underlying semantic categories without domain supervision in a controlled setting, reaching 84.8% expert purity against 60.8% for conventional MoE; on real-world data its experts admit human-interpretable task- and domain-level descriptions aligned with their functional strengths. Targeted interventions on routing weights produce predictable preference changes, showing that these interpretations are behaviorally grounded rather than merely descriptive. Crucially, the disentanglement induced by this objective also makes the experts composable: lightweight router updates yield a 25.81-point improvement in few-shot personalization with 50 examples, roughly 2.7 times the gain of the identical architecture and adaptation procedure applied to entangled experts. Together, these results show that MoE reward models can learn interpretable expert structure, and that the same structure can substantially improve personalization.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2606.04284 [cs.LG] |
| (or arXiv:2606.04284v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2606.04284 arXiv-issued DOI via DataCite |
Submission history
From: Yifan Wang [view email]
[v1]
Tue, 2 Jun 2026 23:19:12 UTC (1,474 KB)
[v2]
Tue, 27 Oct 2026 18:34:57 UTC (1,509 KB)
来源：arXiv cs.AI · arxiv.org