DRMOT：面向 RGBD 指代多目标跟踪的数据集与框架

View PDF HTML (experimental)

Abstract:Referring Multi-Object Tracking (RMOT) aims to track specific targets based on language descriptions, playing a vital role in interactive AI systems such as robotics and autonomous driving. However, existing RMOT methods rely exclusively on 2D RGB observations. Since similar 2D projections can correspond to different 3D configurations, the available observations may not uniquely determine the referred targets. This creates a fundamental mismatch between the task requirements and the available input. To advance research on the DRMOT task, we construct DRSet, a dataset tailored for the DRMOT task. DRSet comprises 2,076 referring expressions over 186 RGB-D source sequences. It features synchronized RGB images and depth maps paired with language descriptions, including explicit depth-dependent descriptions, to systematically evaluate spatial-semantic detection and tracking capabilities. Furthermore, we propose DRTrack, an MLLM-guided depth-referring tracking framework. DRTrack leverages a Geometric-Aware GRPO fine-tuned Multimodal Large Language Model (MLLM) to achieve precise spatial-semantic detection from joint L+RGB+D inputs, while incorporating explicit depth cues to enforce robust trajectory association. Extensive experiments on the DRSet dataset demonstrate the effectiveness of our framework. Our dataset and code will be available at this https URL.

Comments:	Github: this https URL
Subjects:	Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2602.04692 [cs.CV]
	(or arXiv:2602.04692v3 [cs.CV] for this version)
	https://doi.org/10.48550/arXiv.2602.04692 arXiv-issued DOI via DataCite

Submission history

From: Sijia Chen [view email]
[v1] Wed, 4 Feb 2026 15:56:16 UTC (26,470 KB)
[v2] Fri, 6 Feb 2026 04:39:34 UTC (26,470 KB)
[v3] Sun, 11 Oct 2026 05:17:26 UTC (18,603 KB)