arXiv cs.CL· Kunwang Zheng, Xinyu Sun, Yu Cao, Shuocheng Li, Lei Cheng, Ying Liu, Qilong Zhang, Jinlong Chen, Hua Yang, Qianglong Chen·· 3 天前AI 评分24
RELAY：面向评分标准强化学习的自适应交接选择性自蒸馏
RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning
AI 导读
RELAY 提出一种选择性自蒸馏与自适应交接方法，用于基于评分标准的强化学习：Stage 1 用冻结教师模型提供 token 级监督并结合 GRPO，由平台期检测器在蒸馏收益停滞时移除教师，Stage 2 仅用 GRPO 继续训练。
正文 · 原文
Abstract:Rubric-based reinforcement learning grades a response against many written criteria but collapses them into one reward, which discards the detail the rubric carries and gives no GRPO reward gradient when a group's responses tie. A frozen teacher that reads the rubric as privileged context provides token-level supervision, but we observe an early plateau in the gains over GRPO. RELAY uses advantages and rewards to select positive-advantage and low-scoring zero-advantage rollouts, and monitors the distillation loss to decide when to retire the teacher. Stage 1 combines selective distillation from a frozen teacher with GRPO; a plateau detector then removes the teacher, and Stage 2 continues with GRPO alone. Calibrated on one model-domain pair and reused unchanged in three others, the plateau detector fires between steps 122 and 171. RELAY scores highest in-domain among the compared methods in all four settings, ahead of GRPO on RaR-Medicine by 4.62 points with Qwen3-1.7B and 6.02 with Qwen3-4B. On Qwen3-1.7B/Medicine, removing the teacher from a shared checkpoint outperforms keeping, weakening or refreshing it. Distilling selected rollouts covering about half the response tokens also outperforms distilling all rollouts at a matched average coefficient.
| Comments: | 17 pages, 8 figures, 13 tables |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.17623 [cs.LG] |
| (or arXiv:2610.17623v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17623 arXiv-issued DOI via DataCite |
Submission history
From: Kunwang Zheng [view email]
[v1]
Tue, 13 Oct 2026 02:45:54 UTC (1,718 KB)
[v2]
Wed, 14 Oct 2026 05:22:37 UTC (1,718 KB)
来源：arXiv cs.CL · arxiv.org