arXiv cs.CL· Jiarun Shen, Lei Zhao, Xiao Li·· 12 小时前AI 评分22
一个模型两条路径：语言模型单次推理与循环推理的性能平衡
One Model, Two Paths: Balancing Single-Pass and Looped Performance in Language Models
AI 导读
研究提出对预训练 Transformer 复用中间层权重，使同一模型同时支持单次推理与循环推理，但共享权重会让循环训练损害单次推理质量。采用双输出加权损失的联合训练策略可显著减少单次推理性能退化，同时保留循环推理相对普通持续训练的增益；收益主要来自调整单次推理损失权重，冻结原模型的引导还能进一步改善循环输出。按验证集选定的权重，该方法在预设容差内保持单次推理质量并提升循环预测。
正文
Abstract:Repeating Transformer layers can improve language-model performance without adding parameters. A pretrained model can be adapted to support both single-pass and looped inference by reusing middle-layer weights for additional computation. Because both paths share weights, training to improve looped predictions can degrade single-pass quality. We study joint training strategies using weighted losses on both outputs, with guidance from the frozen original model. Joint supervision substantially reduces single-pass degradation compared to looped-only training, while preserving looped gains over ordinary continued training. Most of this benefit comes from adjusting the single-pass loss weight, with guidance from the frozen original model further improving the looped output. With validation-selected weights, our method preserves single-pass quality within a predefined tolerance of ordinary continued training while improving looped predictions.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.33517 [cs.CL] |
| (or arXiv:2610.33517v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33517 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiarun Shen [view email]
[v1]
Sat, 24 Oct 2026 07:50:29 UTC (400 KB)
来源：arXiv cs.CL · arxiv.org