arXiv cs.CL· Md Sakib Ul Islam, S M Mehedi Zaman·· 13 小时前AI 评分29
语言模型新版本未必延续旧版预训练信息披露：41 个模型条目审计
What Carries Forward? Pretraining Disclosure Across Language Model Releases
AI 导读
一项基于文档的审计用 2025 FMTI 的 24 项预训练指标考察 17 个模型家族共 41 个条目，在十个家族的 24 组相邻版本对比中，得分下降 11 组、上升 6 组、持平 7 组，指标净减少 38 项、新增 28 项。OLMo-2 与 OLMo-3 总分相同但各有三项增减，Llama-3.1 总分优于 Llama-2 却丢失两项数据类指标。语料构成与训练算力方面的披露缺口持续存在。
正文 · 原文
Abstract:Understanding how language models were pretrained requires information about their data, processing procedures, and computational resources. However, it remains unclear whether successive releases preserve or expand earlier disclosure. We conduct a document-based audit of 41 model entries from 17 families using 24 pretraining-related indicators from the 2025 Foundation Model Transparency Index (FMTI). We examine overall scores and indicator profiles, tracking gains, losses, and retained disclosures across 24 adjacent comparisons in ten families. Scores decrease in 11 comparisons, increase in six, and remain unchanged in seven, with 38 indicator losses and 28 gains overall. Stable or improving scores can conceal losses: OLMo-2 and OLMo-3 receive identical totals despite three gains and three losses, while Llama-3.1 improves over Llama-2 while losing credit for two data-related criteria. Gaps also persist in corpus composition and training compute. These findings show that later releases do not consistently preserve earlier pretraining disclosure and motivate documentation that explicitly identifies changes and the applicability of inherited information.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.37465 [cs.CL] |
| (or arXiv:2610.37465v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37465 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: S M Mehedi Zaman [view email]
[v1]
Tue, 27 Oct 2026 16:00:49 UTC (60 KB)
来源：arXiv cs.CL · arxiv.org