arXiv cs.CL· Hanyu Yang·· 11 小时前AI 评分31
固定权重下的损失修复：用 Logit 校正缓解语言模型遗忘
Less Loss, Fixed Weights: Logit Corrections for Forgetting
AI 导读
研究检验能否用固定的词表偏置（每个 token 的固定加分或减分）单独或配合温度参数，逆转语言模型微调或继续预训练后的"遗忘"。在 14 对 1B 至 7B 参数的基础模型与衍生模型上，top-4K 偏置的最大损失恢复估计不足 10%，偏置加温度最高可恢复 36%；在数学数据上继续预训练的模型遗忘最严重且几乎无法恢复。
正文 · 原文
Abstract:When a language model is fine-tuned to follow instructions, or further pretrained on a new subject, it often gets worse at predicting standard web text. This is known as "forgetting." We ask whether this loss can be undone by simple, fixed adjustments to the model's final token predictions. Across 14 pairs of base and derived models, from 1B to 7B parameters, we fitted a fixed bonus or penalty for each token (a vocabulary bias), alone or together with a temperature. On text held out from correction fitting, the largest top-4K bias recovery estimate is under 10%, with loss conditional on the retained vocabulary. Combining the bias with a temperature recovers up to 36%. Models further pretrained on math lose the most and recover almost nothing. Finally, in a controlled simulation where a model is damaged at random, with no planted bias or temperature change, the bias-plus-temperature fix still recovers 29%. Most of the real-model loss increase stays out of reach of our restricted fits.
| Comments: | 8 pages, 3 figures, 2 tables. Code and data: this https URL |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.42000 [cs.LG] |
| (or arXiv:2610.42000v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.42000 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hanyu Yang [view email]
[v1]
Fri, 30 Oct 2026 16:44:30 UTC (54 KB)
来源：arXiv cs.CL · arxiv.org