arXiv cs.CL· Ruizhe Shi, Zaiwen Yang, Qiubo Wang, Banghua Zhu, Maryam Fazel·· 2 天前AI 评分22
按梯度范数排序与划分注意力头，实现稀疏持续强化学习微调
Ranking and Partitioning of Attention Heads for Sparse and Continual Reinforcement Learning Fine-tuning
AI 导读
研究通过梯度范数对注意力头排序与划分，实现语言模型的稀疏持续强化学习微调。在数学、指令遵循和编码任务上，微调高排名注意力头持续优于低排名头，且常能匹配更密集的基线；注意力头划分则提升旧任务保留能力，同时保持对新任务的适应。
正文 · 原文
Abstract:We study sparse and continual reinforcement learning fine-tuning of language models through attention-head ranking and partitioning. For sparse learning, we rank attention heads by gradient norm to identify a small subset of heads that can be trained effectively. For continual learning, we partition heads across tasks to reduce cross-task interference and preserve previously learned capabilities. Theoretically, we relate gradient norm to training effectiveness, and further characterize why sparse, separated head updates can remain stable. Empirically, across math, instruction-following, and coding tasks, tuning higher-ranked heads consistently outperforms tuning lower-ranked ones while often matching denser baselines, and head partitioning improves retention while preserving adaptation to new tasks.
| Comments: | 26 pages, comments are welcome |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.21336 [cs.LG] |
| (or arXiv:2610.21336v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21336 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ruizhe Shi [view email]
[v1]
Thu, 15 Oct 2026 06:21:56 UTC (155 KB)
来源：arXiv cs.CL · arxiv.org