arXiv cs.AI· Luyang Zhang, Jialu Wang, Bebei Li·· 3 天前AI 评分29
Pick the Mixture, Not the Samples：面向 LLM 的组合式数据选择
Pick the Mixture, Not the Samples: Combinatorial Data Selection for LLMs
AI 导读
研究者提出组合式数据选择（CDS）框架，将样本评分与子集选择分离，并建模样本间的非加性交互。基于跨域互补性发现，他们提出 Mixture-then-Select 两阶段算法：先决定各域采样数量，再按归因分数排序。该方法在真实指令微调数据池上超越强基线 20% 至 54%，且在不同模型和数据池规模下均保持稳健。
正文 · 原文
Abstract:Data selection for LLM fine-tuning faces two facts about real training: data pools draw from many distinct domains, and evaluation spans diverse tasks. As a result, the same sample helps different tasks differently, and samples interact during training so that a subset's value is not the sum of its parts. Current data-selection methods leave these realities largely unaddressed. We formalize them as Combinatorial Data Selection (CDS), a framework that separates attribution (how samples are scored) from selection (which subset to pick) and models the non-additive interactions among samples. On a real instruction-tuning data pool, we find substantial cross-domain complementarity (joint utility exceeds the sum of singletons), alongside redundancy within domains. Inspired by this, we propose Mixture-then-Select, a two-stage algorithm that first decides how many samples to take from each domain, then ranks samples within each domain by attribution score. Theoretically, the gap to the optimal subset is linear in the selection size and independent of data pool size. Empirically, Mixture-then-Select outperforms strong baselines by 20 to 54%, is robust across attribution methods, and works across different models and data pool sizes. The framework also extends to the data purchasing problem under a budget constraint when sample prices vary. Our findings suggest that allocating selection across domains offers a promising path toward more efficient and effective data selection for modern LLMs.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14028 [cs.LG] |
| (or arXiv:2610.14028v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14028 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Luyang Zhang [view email]
[v1]
Wed, 30 Sep 2026 18:01:37 UTC (214 KB)
来源：arXiv cs.AI · arxiv.org