arXiv cs.AI· Paul Raj, Simon Caldwell, Daniel Whitmore, Mutlu Mete·· 13 小时前AI 评分31
DuplexTune：用耦合状态重放实现 MoE 模型的高效适配
DuplexTune: Coupled-State Replay for Memory-Efficient Mixture-of-Experts Adaptation
AI 导读
DuplexTune 通过可训练宽度桥接连接两条残差流，将耦合隐藏状态重放至预训练注意力与专家模块，实现 MoE 语言模型的内存高效适配。在 Qwen1.5-MoE-A2.7B 上，单张 H800 评测显示峰值内存 38.8 GB，低于带 checkpointing 的监督微调 64.8 GB，吞吐为 25.1 对 19.9 samples/s。
正文 · 原文
Abstract:Sparse expert execution does not remove the stored states needed to adapt a mixture-of-experts language model. We present DuplexTune, a framework for replaying coupled hidden states through pretrained attention and expert modules. Trainable width bridges connect two residual streams to the backbone. Bridge warm-up precedes backbone adaptation, with router weights kept frozen. We analyze the implicit state-recovery equation and identify the convergence and replay conditions needed for reconstruction. On Qwen1.5-MoE-A2.7B, the single-H800 evaluation reports 38.8 GB peak memory, versus 64.8 GB for supervised fine-tuning with checkpointing, and throughput of 25.1 versus 19.9 samples/s. DuplexTune achieves 67.2% MMLU, 75.6% GSM8K, and 7.69 MT-Bench, with a weaker multilingual score than the checkpointed baseline. The results characterize a configuration-dependent tradeoff among state storage, throughput, and adaptation quality.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.36009 [cs.LG] |
| (or arXiv:2610.36009v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36009 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Paul Raj [view email]
[v1]
Mon, 26 Oct 2026 16:17:33 UTC (1,321 KB)
来源：arXiv cs.AI · arxiv.org