arXiv cs.AI· Sheldon Yu, Tong Yu, Nikki Lijing Kuang, Zhendong Chu, Zhenwei Tang, Sungchul Kim, Lina Yao, Julian McAuley, Junda Wu·· 2 天前AI 评分24
RSHD：用失败回放恢复 GRPO 监督信号
Failure Is the Mother of Success: Recovering Supervision with Grounding Hindsight
AI 导读
针对 GRPO 在整组 rollout 全部失败时既无成功样例也无非零组内优势、学不到东西的问题，研究者提出 Replay-Selected Hindsight Distillation（RSHD），让策略在不看参考答案的情况下为每次失败写纠正笔记并带笔记重解，仅当回放成功率高于原组且笔记未泄露答案时才采纳，并将采纳的回放蒸馏回纯提示词策略。
正文 · 原文
Abstract:When every rollout for a problem fails, GRPO receives neither a successful exemplar nor a nonzero group-relative advantage, so it learns nothing from exactly the problems it most needs to learn. The failed reasoning can nevertheless suggest what to try next. We present Replay-Selected Hindsight Distillation (RSHD), which turns failed attempts into an additional, tested source of supervision during GRPO training. The policy writes a corrective note about each failed attempt without seeing a reference answer or solution, then re-solves the problem with the note in context. A note is admitted only if its replays succeed more often than the original group did, after screening for notes that state the answer, and an admitted replay is distilled into the plain-prompt policy. Because admission is decided by fresh successes rather than by the failed rollout itself, the procedure can recover supervision from groups in which every attempt failed, and it applies equally to failures in mixed groups. Only the policy is trained, the only judge is the outcome verifier, and deployment requires no notes or extra inference. Across model families and sizes and two training domains, RSHD consistently improves over GRPO on in-domain benchmarks, and the gains carry over to out-of-domain evaluation sets.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.21018 [cs.AI] |
| (or arXiv:2610.21018v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21018 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zixu Yu [view email]
[v1]
Wed, 14 Oct 2026 23:33:22 UTC (743 KB)
来源：arXiv cs.AI · arxiv.org