arXiv cs.CL· Mengyao Lyu, Yan Li, Huasong Zhong, Hui Chen, Jungong Han, Guiguang Ding, Zhenheng Yang·· 13 小时前AI 评分38
SSR：面向百万级多模态指令微调的极简数据选择策略
An Embarrassingly Simple Data Selection Strategy for Million-Scale Multi-Modal Instruction-Tuning
AI 导读
研究者提出 SSR 数据选择策略，用交互风格表征结构多样性、将数据质量分解为能力维度得分，并把选择转化为能力—风格网格上的子模覆盖最大化问题。在 LLaVA-1.5 与 LLaVA-OneVision 两套架构、14 个多模态基准上，SSR 仅用 2.6M 数据池的 30% 即达到全量数据 99.1% 的性能，是首个在百万级规模验证的数据选择方法，其画像模型与子集可跨数据池和架构免训练迁移。
正文 · 原文
Abstract:Recent research demonstrates that pretrained large language models (LLMs) require minimal but carefully curated data for instruction-tuning. Building on LLMs, multi-modal models further contend with far larger data volumes, heterogeneous sources, and cross-modal alignment, amplifying both the necessity and the complexity of data selection. However, existing methods rely on deeply extracted embeddings, gradients or narrow quality scores that are time-consuming to compute and limited in expressiveness, frequently failing to surpass a simple random-sampling baseline. To overcome these limitations, we step back from complex embedding spaces and target the fundamental objective of instruction tuning: how and what knowledge should be elicited from the pretrained model and exposed to users. We leverage interaction \underline{S}tyles as a proxy for structural diversity, and decompose data quality into capability-specific \underline{S}cores, yielding an explicit yet \underline{R}ich (SSR) profile for each sample. We further cast the selection as submodular coverage maximization over the capability--style grid, providing an approximation guarantee. SSR consistently delivers superior performance across two distinct architecture stacks, LLaVA-1.5 (Vicuna + CLIP) and LLaVA-OneVision (Qwen2 + SigLIP), two large-scale data pools, six data budget settings, and 14 multi-modal benchmarks. To our knowledge, SSR is the first data selection method validated at million-scale, achieving 99.1% of full-data performance using only 30% of a 2.6M pool. Furthermore, the SSR profile models and selected subsets transfer training-free across data pools and model architectures with capability extensibility.
| Comments: | Accepted to EMNLP 2026 Main Conference (Oral) |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2503.13383 [cs.CV] |
| (or arXiv:2503.13383v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2503.13383 arXiv-issued DOI via DataCite |
Submission history
From: Mengyao Lyu [view email]
[v1]
Mon, 17 Mar 2025 17:11:22 UTC (4,281 KB)
[v2]
Tue, 27 Oct 2026 14:31:21 UTC (3,061 KB)
来源：arXiv cs.CL · arxiv.org