arXiv cs.AI· Nikolaos Gkalelis, Vasileios Mezaris·· 13 小时前AI 评分29
LLaVA-CKD：面向视觉语言模型的自底向上级联知识蒸馏框架
LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Models
AI 导读
研究者提出 LLaVA-CKD 自底向上级联知识蒸馏框架，在容量差距过大的教师模型与学生模型之间引入中间容量教师，逐级缩小差距以改善知识迁移。该方法基于 LLaVA 构建模型，在七个公开 VQA 基准上评测，性能与近期紧凑型 VLM 相当，并附有理论分析说明渐进缩小师生差距为何有效。级联蒸馏源码与训练模型已开源。
正文 · 原文
Abstract:Large Vision-Language Models (VLMs) are successful in addressing a multitude of vision-language understanding tasks, such as Visual Question Answering (VQA), but their memory and compute requirements remain a concern for practical deployment. A promising class of techniques for mitigating this concern is Knowledge Distillation, where knowledge from a high-capacity Teacher network is transferred to a considerably smaller Student network. However, the capacity gap between the two networks is both a blessing and a curse: the smaller the Student network, the better its efficiency, and the stronger (typically, also larger) the Teacher, the more knowledge it carries; yet, beyond a point, the larger capacity gap between the two leads to worse knowledge transfer. To counter this effect, we propose a bottom-up cascaded knowledge distillation (CKD) framework. Instead of treating knowledge transfer as an activity involving one high-capacity Teacher (or an ensemble of such), inspired by human formal education systems, we introduce one (potentially, more) additional Teacher(s) of intermediate capacity that gradually bring the Student network to the next level, where the next (higher-capacity) Teacher can take over. We provide a theoretical analysis to motivate why progressive reduction of the Teacher-Student gap may improve distillation. We apply the proposed framework on models built upon the LLaVA methodology and evaluate the derived models on seven standard, publicly available VQA benchmarks, demonstrating competitive performance with recent compact VLMs. Source code for cascaded distillation and trained models are available at this https URL.
| Comments: | IEEE Access, vol.14, 2026 |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.10641 [cs.CV] |
| (or arXiv:2605.10641v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2605.10641 arXiv-issued DOI via DataCite |
|
| Related DOI: | https://doi.org/10.1109/ACCESS.2026.3744200
DOI(s) linking to related resources |
Submission history
From: Vasileios Mezaris [view email]
[v1]
Mon, 11 May 2026 14:28:44 UTC (76 KB)
[v2]
Tue, 20 Oct 2026 11:01:38 UTC (8,084 KB)
来源：arXiv cs.AI · arxiv.org