arXiv cs.AI· Amir Mohammad Karimi Mamaghan, Ferdinand Kapl, Andrea Dittadi, Karl Henrik Johansson, Kaitlin Maile, Johannes von Oswald, Stefan Bauer·· 3 天前AI 评分23
渐进式深度增长如何高效预训练自监督视觉 Transformer
Growing to See: Efficient Pretraining of Self-Supervised Vision Transformers
AI 导读
研究系统考察了渐进式深度增长在 MAE 和 DINOv3 两类自监督视觉 Transformer 中的效果：在全局与稠密任务上，该方法将训练 FLOPs 降低 13.9%–24.8%，ImageNet 线性探测准确率最高提升 3.36 个百分点。增长还改善了中间层表征，并重塑内部计算模式——MAE 中出现全局与局部注意力反复交替，自蒸馏 DINOv3 中局部到全局注意力的过渡提前出现。
正文 · 原文
Abstract:Progressive depth growth builds large networks by starting with shallow models and inserting layers during training. Recent advances in large language models have renewed interest in this longstanding approach, demonstrating benefits for training efficiency and reasoning alongside changes in internal computations. However, how growing affects representation quality and computational patterns in self-supervised vision transformers remains unclear. In this work, we systematically investigate progressive depth growth in two popular self-supervised families, namely masked autoencoders (MAE) and DINOv3, and analyze how growing affects downstream performance, intermediate representations, and internal computations. Across global and dense tasks, we show that growing improves downstream performance while reducing training FLOPs by 13.9--24.8%, with ImageNet linear-probe accuracy gains of up to 3.36 percentage points. Growing also improves downstream performance at intermediate layers and reshapes internal computational patterns, introducing recurring alternations between global and localized attention in MAE and earlier local-to-global attention progression in self-distilled DINOv3. Together, these findings establish progressive depth growth as an effective and efficient pretraining strategy for improving downstream performance across global and dense visual tasks with less computation, and advance our understanding of how growing shapes visual computation.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.16817 [cs.CV] |
| (or arXiv:2610.16817v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.16817 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Amir Mohammad Karimi Mamaghan [view email]
[v1]
Mon, 12 Oct 2026 14:01:13 UTC (4,756 KB)
来源：arXiv cs.AI · arxiv.org