arXiv cs.CL· Yudong Zhang, Wei Ding, Kaili Zheng, Yuchen Ding, Zhenyu Zhang·· 12 小时前AI 评分31
Accordion：基于固定前缀层循环的弹性深度推理模型
Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping
AI 导读
Accordion 是一个基于混合专家（MoE）的模型，通过固定前缀层循环支持五种推理预算，最低预算保留未循环模型。经过 3.6T token 预训练后，增加推理深度使 Accordion-6B-A0.5B 在 37 项任务上的五类宏平均提升 3.01 个百分点，中间预算以一半的额外层执行量获得 89% 的增益。最终模型在各领域任务上超过对应领域专家，并在所有预算下保持较强的指令遵循能力。
正文 · 原文
Abstract:Large language models serve applications with diverse inference budgets, but training and maintaining separate models for each budget incurs substantial costs. Recent Looped Transformers enable flexible inference computation through parameter reuse, but maintaining strong performance across different inference budgets remains an open challenge. We introduce Accordion, a mixture-of-experts-based model that supports five inference budgets through fixed-prefix layer looping. Each budget follows a pre-defined execution path, while the minimum budget retains the unlooped model. Joint pretraining samples one budget per micro-batch and optimizes its next-token prediction loss without additional cross-budget consistency objectives. We further preserve these inference budgets through supervised fine-tuning and reinforcement learning, where parameter-group merging and budget-matched multi-teacher on-policy distillation integrate three domain experts into one model. After 3.6T tokens of pretraining, increasing inference depth improves Accordion-6B-A0.5B's five-category macro-average across 37 tasks by 3.01 percentage points, with the middle budget capturing 89% of this gain using half as many additional layer executions. The final model surpasses each domain expert on its corresponding task-and-budget average, while achieving strong instruction-following performance across all budgets and retaining depth-dependent gains in mathematical reasoning and code generation.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.32492 [cs.LG] |
| (or arXiv:2610.32492v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32492 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yudong Zhang [view email]
[v1]
Fri, 23 Oct 2026 11:07:45 UTC (1,172 KB)
来源：arXiv cs.CL · arxiv.org