arXiv cs.CL· Aryan Das, Vinay Kumar Verma, Aakansha Mishra·· 11 小时前AI 评分31
Muon-RT：复用教师轨迹实现无动量 Muon 的 LLM 自蒸馏方法
MUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation
AI 导读
Muon-RT 通过复用教师与学生之间的位移来消除隐藏权重矩阵的动量缓冲区，从而降低持续训练内存。在 Qwen2.5-7B-Instruct 上用 SDFT 训练时，其平均准确率比 AdamW 高 2.6 分且持续内存少 42%，比带动量的 Muon 高 5.4 分且内存少 27%；在 Olmo-3-7B-Instruct 上用 SDPO 训练时比 AdamW 高 2.0 分。
正文 · 原文
Abstract:Self-distillation methods for large language models with an exponential moving average (EMA) teacher already retain a history of student updates. Muon nevertheless stores a separate momentum buffer, increasing persistent training memory. We propose Muon with Reused Teacher (Muon-RT), which reuses the teacher-student displacement to eliminate momentum buffers for hidden weight matrices. We show that this displacement encodes an exponentially weighted history of applied updates and smooths temporally uncorrelated noise under a local model. Muon-RT removes its radial component, normalizes it spectrally, and mixes it with the current gradient after Newton-Schulz (NS) orthogonalization. Under explicit regularity and alignment assumptions, we establish linear contraction of an expected objective-gap bound toward a residual error level. With Self-Distillation Fine-Tuning (SDFT) on Qwen2.5-7B-Instruct, Muon-RT improves average accuracy over AdamW by 2.6 points with 42% less persistent memory, and over momentum Muon by 5.4 points with 27% less persistent memory. With Self-Distillation Policy Optimization (SDPO) on Olmo-3-7B-Instruct, it improves average accuracy over AdamW by 2.0 points.
| Comments: | Under Submission |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.41104 [cs.LG] |
| (or arXiv:2610.41104v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41104 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Vinay Verma Kumar [view email]
[v1]
Fri, 30 Oct 2026 00:50:49 UTC (261 KB)
来源：arXiv cs.CL · arxiv.org