arXiv cs.AI· Shengtao Wu, Zichao Dong, Weixuan Qiao, Xiaolu Liu, Xin Zhan, Junbo Chen, Jianke Zhu·· 1 天前AI 评分34
从基础 VLM 到通用驾驶 VLM：一种三元组增量训练方法
From Foundation VLM to Versatile Driving VLM: A Triplet Incremental Training Approach
AI 导读
研究提出 Triplet Incremental Training 框架，无需改造架构即可将通用 VLM 适配自动驾驶，通过（图像、深度、QA 对）三元组进行三流训练，部署时丢弃辅助深度分支，实现零推理开销。该方法在 3D 空间感知、驾驶语言推理、端到端规划和通用多模态理解四个基准上超越现有驾驶 VLM，3D 感知媲美领域专家模型，并保留原有通用推理能力。论文已被 CoRL 2026 接收。
正文 · 原文
Abstract:Adapting general Vision-Language Models (VLMs) to Autonomous Driving (AD) is thought to require heavy 3D spatial priors or intrusive architectural overhauls to compensate for a lack of spatial grounding. In this work, we present a counterintuitive pilot study: off-the-shelf general VLMs already exhibit remarkable adaptability across diverse driving tasks, while precise metric depth remains the primary missing link, a bottleneck readily addressable via onboard sensors and depth foundation models. To this end, we introduce the Triplet Incremental Training Framework, a non-intrusive paradigm that seamlessly instills 3D geometric awareness into a general VLM backbone while fully preserving its open-world capabilities. Driven by automatically constructed (image, depth, QA pair) triplets, our framework operates via a unified tri-stream training pipeline: an unfrozen primary VLM stream to adapt to driving tasks, an auxiliary depth stream that guides the visual backbone to implicitly imagine 3D scene geometry via explicit geometric constraints, and a frozen twin VLM stream employing logit-level KL distillation to prevent semantic drift. Crucially, the auxiliary depth branch is discarded during deployment, ensuring the enhanced geometric awareness comes with zero inference overhead. We evaluate our framework across four major benchmarks targeting four diverse capabilities spanning 3D spatial perception, driving-oriented language reasoning, end-to-end planning, and general multimodal understanding. Our approach outperforms state-of-the-art driving VLMs, rivals domain-specific expert models in 3D perception, and successfully retains its original general reasoning capabilities. This demonstrates that Triplet Incremental Training offers a unified, deployment-ready paradigm for transforming general foundation models into robust driving intelligences.
| Comments: | 13 pages, 3 figures, 7 tables. Accepted at the 10th Conference on Robot Learning (CoRL 2026) |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.16727 [cs.CV] |
| (or arXiv:2611.16727v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.16727 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shengtao Wu [view email]
[v1]
Thu, 12 Nov 2026 12:39:25 UTC (21,173 KB)
来源：arXiv cs.AI · arxiv.org