arXiv cs.CL· Ellen Su, Andres Potapczynski, Shikai Qiu, Edward Hughes, Andrew Gordon Wilson·· 12 小时前AI 评分31
用 Epiplexity 引导数据选择与生成，提升分布外泛化能力
Epiplexity Guided Data Selection and Generation for Out-of-Distribution Generalization
AI 导读
研究者将 Epiplexity（计算受限学习器可从数据中提取的结构信息量）用作在线训练信号，提出两种方法：EpiSelect 通过拟合自然数据域训练损失曲线的缩放定律预测 Epiplexity 增益，自适应调整各域采样权重；EpiGen 在无域结构的合成数据上用 REINFORCE 奖励生成器提升学习器合成数据的 Epiplexity 变化。
正文 · 原文
Abstract:Modern systems are increasingly expected to transfer across tasks not specified during training. What data facilitates generalization to unanticipated settings? One hypothesis is that data with more structural information could contain shared circuits and subprograms that may be recycled in a wider array of downstream settings. Epiplexity, a recently proposed measure of the structural information a compute-bounded learner can extract from data, offers a mechanism to reason about this relationship. As existing estimators for epiplexity only operate on data already seen during training, we show how to operationalize epiplexity as an online training signal by greedily maximizing its stepwise gain in two settings. For data selection, we fit scaling laws to the training loss curves of natural data domains to predict the expected epiplexity gain as a function of training tokens, and use this signal to adaptively determine the sampling weights over domains during training. For synthetic data generation, where no domain structure exists, we reward a generator with the change in epiplexity of a learner's synthetic training data and use REINFORCE policy gradients to guide the generator toward an epiplexity-maximizing distribution. In both cases, we find higher epiplexity predicts improved downstream performance on zero-shot and fine-tuning tasks, supporting the hypothesis that data rich in structural information yield representations that better transfer across domains.
| Comments: | Code available for EpiSelect (this https URL) and EpiGen (this https URL) |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2608.11746 [cs.LG] |
| (or arXiv:2608.11746v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2608.11746 arXiv-issued DOI via DataCite |
Submission history
From: Ellen Su [view email]
[v1]
Wed, 12 Aug 2026 07:38:58 UTC (854 KB)
[v2]
Wed, 4 Nov 2026 02:17:42 UTC (324 KB)
来源：arXiv cs.CL · arxiv.org