arXiv cs.CL· Nam H. Le·· 12 小时前AI 评分42
模型复制知识该由谁买单？研究将模型坍缩解读为 Rogers 悖论
Who Pays for the Knowledge Models Copy? Model Collapse as Rogers' Paradox
AI 导读
研究把生成模型的模型坍缩重新解读为 Rogers 悖论：依赖复制的一代并不比从不复制更好，因为副本会过时，除非有人持续自主学习。仅用纯合成数据拟合的 Rogers 模型，能在几个百分点误差内预测从词三元组到预训练 Transformer 的语言模型在新鲜数据不同混合比例下的稀有知识保留率。
正文 · 原文
Abstract:Generative models trained on the output of earlier models lose knowledge across generations, rare knowledge first. The remedies that work all bring in information from outside the loop, and the theory of model collapse treats that supply as given: a fixed archive, drawn from a fixed distribution. We argue that the supply is the problem, and that cultural evolution has the theory of it. We read model collapse as Rogers' paradox: a population that comes to rely on copying ends up no better off than one that never copied, because copies go stale unless someone keeps learning for themselves. We test the identification in three steps. First, Rogers' model, fitted only to fully synthetic training, predicts to within a few percentage points how much rare knowledge language models, from word trigrams to a pretrained transformer, retain under other mixtures of fresh data. Second, when the world itself changes, a lineage that copies its own past needs more fresh data to keep up the faster the world changes, as Rogers' model says it must and as analyses that assume a fixed target cannot; a lineage that restarts every generation from fixed pretrained weights is largely insulated, because those weights are knowledge that is never copied. Third, we close the loop: when the people who write a model's training data can ask the model instead of finding things out, the supply of fresh data becomes endogenous. At equilibrium, the model is exactly as reliable as it is costly for people to find out for themselves, the population is no better off than without the model, and every increase in the share of model-written text must be paid for by more people learning for themselves. Users who check what they are told and relearn when it is wrong break the paradox. Filters that select plausible text are conformist and accelerate collapse; only checks that carry new information act as critical learning.
| Comments: | 13 pages, 5 figures. Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL); Populations and Evolution (q-bio.PE) |
| Cite as: | arXiv:2611.06022 [cs.LG] |
| (or arXiv:2611.06022v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06022 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nam H. Le [view email]
[v1]
Wed, 4 Nov 2026 06:19:55 UTC (85 KB)
来源：arXiv cs.CL · arxiv.org