arXiv cs.CL· Lucas Bandarkar, Alan Ansell, Trevor Cohn·· 11 小时前AI 评分34
利用跨语言不一致性定位 MoE 大语言模型中的知识
Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistency
AI 导读
研究者提出用跨语言不一致性作为 MoE 大语言模型的可解释性工具，通过对比模型在不同语言下答对与答错时的路由差异来定位知识。该方法先跨多语言提问生成"成功"与"失败"激活桶，再对 MoE 路由器 logits 做统计对比分析，找出对特定知识关键的专家。实验显示，仅停用约 6000 个专家中的 20 个，模型在超过 40% 的情况下就无法正确作答。
正文 · 原文
Abstract:Modern LLMs continue to exhibit significant variance in behavior across languages, such as being able to recall factual information in some languages but not others. While typically studied as a problem to be mitigated, in this work, we propose leveraging this cross-lingual inconsistency as a tool for interpretability in mixture-of-experts (MoE) LLMs. Our knowledge localization framework contrasts routing for sets of languages where the model correctly recalls information from languages where it fails. This allows us to isolate model components that play a functional role in answering about a piece of knowledge. Our method proceeds in two stages: (1) querying the model with difficult factual questions across a diverse set of languages to generate "success" and "failure" activation buckets and then (2) applying a statistical contrastive analysis to the MoE router logits to identify experts important for knowledge. To validate the necessity of this small number of experts for answering a knowledge question, we deactivate them and re-ask the question. We find that despite only deactivating about 20 out of 6000 experts, the model no longer answers correctly in over 40% of cases. Generally, this method provides a realistic and scalable knowledge localization approach to address increasingly complex LLMs.
| Comments: | NeurIPS 2026, this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2603.17102 [cs.CL] |
| (or arXiv:2603.17102v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2603.17102 arXiv-issued DOI via DataCite |
Submission history
From: Lucas Bandarkar [view email]
[v1]
Tue, 17 Mar 2026 19:48:44 UTC (1,216 KB)
[v2]
Fri, 6 Nov 2026 16:33:23 UTC (351 KB)
来源：arXiv cs.CL · arxiv.org