arXiv cs.CL· Junrong Tang, Jianghui Wang, Hao Kong, Yang Lin, Yibo Yang·· 11 小时前AI 评分35
HyperRepair：恢复微调语言模型遗忘的通用能力
HyperRepair: Recovering Forgotten General Capabilities of Fine-Tuned Language Models
AI 导读
HyperRepair 框架可在仅付出推理成本的情况下恢复微调语言模型被削弱的通用能力，同时保留领域专精。它从源模型上的修复过程学习超网络，利用激活指纹为未见适配领域的模型生成定制 Repairing Side Net（RSN），无需目标领域样本或额外优化。
正文 · 原文
Abstract:Fine-tuning improves a language model's performance in a target domain but can weaken its general capabilities. Existing methods often face a trade-off between general capabilities and domain specialization, making it difficult to preserve both. We introduce HyperRepair, a framework for recovering general capabilities while retaining domain specialization at only inference cost. To make recovery generalizable across models sharing a pretrained backbone, HyperRepair learns a hypernetwork from repairs trained on source models. The hypernetwork uses activation fingerprints that summarize changes from the pretrained backbone to generate a tailored Repairing Side Net (RSN), whose prompt-conditioned gate applies corrections selectively. This enables recovery on models from unseen adaptation domains without target-domain examples or additional optimization. We evaluate HyperRpair across two model families, 12 LoRA adaptation datasets, and 15 general benchmarks. On the default split, HyperRepair achieves 93.4% of the base models' average general score while retaining 99.5% of the fine-tuned models' average domain score.
| Comments: | 29 pages, 5 figures |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.41147 [cs.CL] |
| (or arXiv:2610.41147v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41147 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Junrong Tang [view email]
[v1]
Fri, 30 Oct 2026 02:09:34 UTC (982 KB)
来源：arXiv cs.CL · arxiv.org