arXiv cs.CL· Xiang Li, Chengting Yu, Jungang Li, Yilun Zheng, Junzhuo Li, Xingyao Zhang, Wenbo Su, Xuming Hu·· 2 天前AI 评分31
HiLo-RD：面向掩码扩散语言模型的跨预算自蒸馏方法
HiLo-RD: Cross-Budget Self-Distillation for Masked Diffusion Language Models
AI 导读
HiLo-RD 通过将大窗口教师状态与小窗口裁剪视图配对，并在提交位置匹配隐藏表示，提升掩码扩散语言模型在有限生成预算下的表现。在 GSM8K 上准确率提升 9.4 个百分点，平均可读长度缩短约 25%，且无需代码领域训练即可迁移到代码生成。该方法无需外部教师模型，同时引导 token 预测与提交顺序。
正文 · 原文
Abstract:We propose HiLo-RD (High-to-Low Budget Representation Distillation) to improve masked diffusion language models (MDLMs) under a limited generation budget. Unlike autoregressive models, MDLMs choose both what to generate and where to commit it. This creates two coupled challenges: predicting useful content before reasoning is complete, and deciding when to fix answer tokens that shape later predictions. We identify budget-induced premature commitment: even when a correct solution fits, the model can fix answer fragments near the window boundary before resolving intermediate steps. These fragments become context for later predictions and redirect subsequent reasoning. Existing post-training methods do not directly transfer the stronger trajectories that the same model follows with a larger window. HiLo-RD addresses this gap by (1) pairing large-window teacher states with cropped small-window views and (2) matching their hidden representations at committed positions. This supervision guides both token prediction and commitment order without an external teacher. Across mathematics benchmarks, HiLo-RD improves low-budget accuracy and shortens responses; these gains extend to code generation without code-domain training. On GSM8K, accuracy rises by 9.4 percentage points as mean readable length falls by about 25%. Attention diagnostics and fixed-budget order interventions support the proposed mechanism. Together, these results show that large-window reasoning can improve small-window decoding.
| Comments: | 20 pages, 9 figures, 3 tables; Under review |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.36901 [cs.LG] |
| (or arXiv:2610.36901v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36901 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xiang Li [view email]
[v1]
Tue, 27 Oct 2026 09:34:08 UTC (546 KB)
来源：arXiv cs.CL · arxiv.org