TOPS：基于构建 Token 最优保留集的第一性原理视觉 Token 剪枝，加速 MLLM 推理

Authors:Tinghao Wang, Yichen Guo, Rui Huang, Junchi Yao, Qizhe Zhang, Zheng Lu, Yuan Zhang, Jiajun Cao, Zhirong Shen, Yaosong Du, Lijie Hu, Shanghang Zhang

View PDF HTML (experimental)

Abstract:Multimodal large language models (MLLMs) have achieved strong multimodal reasoning capabilities, but their efficiency is limited by the large number of visual tokens, which introduces substantial computational overhead. Visual token pruning offers a natural solution, yet existing methods are imperfect: attention-based criteria tend to retain redundant tokens, while diversity-based criteria are often agnostic to user instructions. Even methods that combine multiple criteria still lack a principled formulation of the intrinsic objective of token pruning. In this paper, we revisit visual token pruning from a first-principles perspective and formulate it as constructing Token Optimal Preservation Sets. Through a top-down information-theoretic analysis, we identify three fundamental principles for effective token selection: Task Relevance, Information Coverage, and Semantic Diversity. Based on these principles, we propose TOPS, a training-free and model-agnostic pruning module that can be applied to various MLLMs. Extensive experiments on 7 MLLM backbones and 14 benchmarks demonstrate that TOPS outperforms prior methods under diverse pruning settings. Notably, on LLaVA-NeXT, TOPS removes 77.8% of visual tokens while preserving 100.0% and 100.6% performance on its 7B and 13B models, respectively, suggesting that pruning redundant visual tokens can sometimes mitigate hallucination and inspire future lightweight MLLM design. Our code is available at this https URL.

Submission history

From: Tinghao Wang [view email]
[v1] Thu, 25 Jun 2026 15:29:37 UTC (11,044 KB)
[v2] Wed, 23 Sep 2026 10:09:35 UTC (11,044 KB)
[v3] Thu, 15 Oct 2026 12:34:36 UTC (11,043 KB)