arXiv cs.CL· Jiabin Fan, Guoqing Luo, Michael Bowling, Lili Mou·· 12 小时前AI 评分24
KETCHUP：面向序列知识蒸馏的 K 步回报估计方法
KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation
AI 导读
KETCHUP 提出一种用于文本生成任务中基于强化学习（RL）的知识蒸馏（KD）的 k 步回报估计方法，通过 Bellman 最优方程推导多步回报。理论分析表明该形式可降低梯度估计方差，在学生模型规模较大时改善 RL 优化。在三个文本生成任务上，该方法在标准任务指标和基于 LLM 的评测中均取得更优表现，论文已被 EACL 2026 接收。
正文 · 原文
Abstract:We propose a novel k-step return estimation method (called KETCHUP) for Reinforcement Learning(RL)-based knowledge distillation (KD) in text generation tasks. Our idea is to induce a K-step return by using the Bellman Optimality Equation for multiple steps. Theoretical analysis shows that this K-step formulation reduces the variance of the gradient estimates, thus leading to improved RL optimization especially when the student model size is large. Empirical evaluation on three text generation tasks demonstrates that our approach yields superior performance in both standard task metrics and large language model (LLM)-based evaluation. These results suggest that our K-step return induction offers a promising direction for enhancing RL-based KD in LLM research.
| Comments: | Accepted to EACL 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2504.19024 [cs.CL] |
| (or arXiv:2504.19024v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2504.19024 arXiv-issued DOI via DataCite |
|
| Journal reference: | Findings of the Association for Computational Linguistics: EACL 2026 |
| Related DOI: | https://doi.org/10.18653/v1/2026.findings-eacl.39
DOI(s) linking to related resources |
Submission history
From: Jiabin Fan [view email]
[v1]
Sat, 26 Apr 2025 21:21:04 UTC (307 KB)
[v2]
Mon, 19 Oct 2026 21:36:25 UTC (273 KB)
来源：arXiv cs.CL · arxiv.org