arXiv cs.AI· Mark Marosi, Peter Antal·· 10 小时前AI 评分58
研究:模型的通用进步可替代特定训练数据
General progress substitutes for specific training data in language models
AI 导读
arXiv 论文研究显示,模型在其余训练数据上的通用进步可以替代被刻意剔除的特定训练数据。作者在国际象棋、黑白棋、分子字符串、括号语言和网页文本等任务中剔除单项技能的训练信号,发现较大模型对该技能的延迟可由较小模型预测;一个从未显式训练应对将军的国际象棋模型,其合法应对排名几乎与训练过的模型相当。
正文 · 原文
Abstract:Language models acquire their capabilities from training data, and developers withhold data during training to keep models from learning capabilities that could be misused. How long this delays a capability, and how much data is needed to acquire it later, is unknown. Here we show that the general progress a model makes on the rest of its training data substitutes for the data that was withheld. We withheld the training signal for individual skills in models of chess, Othello, molecular strings, bracket languages and web text. The delay of a withheld skill in larger models was predicted from smaller ones. A chess model never explicitly trained to respond to check ranked the legal responses almost as well as a model that was. When models were then trained on the withheld data, they acquired the skill with a small fraction of the data needed from initialization, and this fraction decreased as the models progressed. A control rule of equal difficulty was learned much more slowly. A programming language withheld from the training text of a 7-billion-parameter model was regained in the same way. Data filtering therefore delays a capability, and the data required to regain it decreases as models improve.
| Comments: | 49 pages, 16 figures, 6 tables. Code: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.24844 [cs.CL] |
| (or arXiv:2611.24844v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24844 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Márk Marosi [view email]
[v1]
Wed, 18 Nov 2026 18:07:55 UTC (1,377 KB)
来源:arXiv cs.AI · arxiv.org