arXiv cs.CL· Mingzi Cao, Xi Wang, Nikolaos Aletras·· 9 天前AI 评分22
最优传输深度扩展 OT-DUS：用最优传输对齐神经元构建新层
Optimal Transport Depth Up-Scaling
AI 导读
针对深度扩展中复制或平均基座层导致神经元排列错位、损害性能的问题，研究者提出 OT-DUS，利用最优传输理论在相邻基座层间逐模块对齐并融合功能对应的神经元来构建新层。在持续预训练和有监督微调下，该方法在不同模型规模和模型族上于通用与专业领域均优于现有方法。分析还发现，在更高位置插入新层不仅性能更强，训练效率也更高。
正文 · 原文
Abstract:Pre-training Large Language Models (LLMs) from scratch at larger scales yields remarkable performance but incurs substantially high training costs. Depth up-scaling provides an efficient alternative by inserting new layers into a pre-trained LLM, avoiding training from scratch. However, most existing methods copying or averaging base layers for new layer, which misalign functionally corresponding neurons, leading to neuron permutation mismatch that harms performance. To address this issue, we propose Optimal Transport Depth Up-Scaling (OT-DUS), which leverages Optimal Transport (OT) theory to align and fuse functionally corresponding neurons module by module in adjacent base layers for new layer construction. OT-DUS achieves better overall performance in both general and specialized domains than existing methods for continual pre-training and supervised fine-tuning across different model sizes and model families. Our analysis of insertion strategies further finds that inserting new layers at higher positions yields not only stronger performance but also improved training efficiency.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2508.08011 [cs.CL] |
| (or arXiv:2508.08011v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2508.08011 arXiv-issued DOI via DataCite |
Submission history
From: Mingzi Cao [view email]
[v1]
Mon, 11 Aug 2025 14:15:33 UTC (336 KB)
[v2]
Wed, 7 Oct 2026 23:52:52 UTC (342 KB)
来源：arXiv cs.CL · arxiv.org