TempRMOT：面向多样化动态表达的多目标指代跟踪框架与 Refer-KITTI-V2 基准

View PDF HTML (experimental)

Abstract:Referring understanding is a fundamental task that bridges natural language and visual content by localizing objects described in free-form expressions. However, existing works are constrained by limited language expressiveness, lacking the capacity to model object dynamics in spatial numbers and temporal states. To address these limitations, we introduce a new and general referring understanding task, termed referring multi-object tracking (RMOT). Its core idea is to employ a language expression as a semantic cue to guide the prediction of multi-object tracking, comprehensively accounting for variations in object quantity and temporal semantics. Along with RMOT, we introduce a RMOT benchmark named Refer-KITTI-V2, featuring scalable and diverse language expressions. To efficiently generate high-quality annotations covering object dynamics with minimal manual effort, we propose a semi-automatic labeling pipeline that formulates a total of 9,758 language prompts. In addition, we propose TempRMOT, an elegant end-to-end Transformer-based framework for RMOT. At its core is a query-driven Temporal Enhancement Module that represents each object as a Transformer query, enabling long-term spatial-temporal interactions with other objects and past frames to efficiently refine these queries. TempRMOT achieves state-of-the-art performance on both Refer-KITTI and Refer-KITTI-V2, demonstrating the effectiveness of our approach. The source code and dataset is available at this https URL.

Subjects:	Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)
Cite as:	arXiv:2406.05039 [cs.CV]
	(or arXiv:2406.05039v3 [cs.CV] for this version)
	https://doi.org/10.48550/arXiv.2406.05039 arXiv-issued DOI via DataCite

Submission history

From: Yani Zhang [view email]
[v1] Fri, 7 Jun 2024 16:02:10 UTC (19,306 KB)
[v2] Mon, 27 Oct 2025 14:22:30 UTC (3,799 KB)
[v3] Fri, 16 Oct 2026 14:34:19 UTC (5,100 KB)