arXiv cs.AI· Fei Wang, Baochun Li, Bo Li·· 1 天前AI 评分31
微调 MoE 大语言模型中的可恢复专家利用特征研究
Characterizing Recoverable Expert-Utilization Signatures in Fine-Tuned Mixture-of-Experts LLMs
AI 导读
研究发现微调后的 MoE 大语言模型存在"结构性记忆"漏洞：模型不仅记住训练数据，还记住处理数据所用的内部路由路径，形成独特的路由指纹，且仅需少量样本即可出现。攻击者无需了解微调数据或流程，仅凭路由与输出 logits 就能通过模型无条件生成隐秘恢复这些专家路径。传统成员推理指标无法识别此类记忆数据，作者提出结合参考模型尾部似然与复杂度感知归一化的复合指标以提升检测能力。
正文 · 原文
Abstract:As Mixture-of-Experts (MoE) large language models are increasingly deployed in distributed environments, from federated learning to edge-cloud systems, their architecture harbors an underexplored landscape of security vulnerabilities. Moving beyond textual analysis, we uncover structural memorization: a novel vulnerability where a fine-tuned model memorizes not only its training data but also the internal pathway used to process it, creating a characteristic routing fingerprint unique to the training process. We provide comprehensive empirical evidence that structural memorization emerges consistently even from a few samples, and that its intensity is directly influenced by the choice of trainable modules.
We show that these memorized expert pathways can be stealthily recovered through the model's unconditional generation, using only routing and output logits, with no knowledge of the fine-tuning data or procedure. In distributed deployments, this vulnerability is amplified as routing information is exposed at multiple trust boundaries, including federated aggregation points, edge-cloud interfaces, and multi-tenant monitoring systems. Recovery exposes a critical blind spot: traditional membership inference metrics, viewed through the lens of expert routing, fail to identify memorized data corresponding to these structural pathways. To close this gap, we devise composite metrics combining reference model tail likelihood with complexity-aware normalization, significantly improving detection of memorized routing structures. This enables adversaries to pinpoint critical experts shaped by fine-tuning, whether dominant or dormant, and execute targeted attacks such as router manipulation or backdoor injection in realistic distributed settings where attackers lack training data but can observe routing patterns across system components.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR) |
| Cite as: | arXiv:2610.41039 [cs.LG] |
| (or arXiv:2610.41039v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41039 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Fei Wang [view email]
[v1]
Thu, 29 Oct 2026 22:30:02 UTC (773 KB)
来源：arXiv cs.AI · arxiv.org