arXiv cs.AI· Yutong Chen, Zhichao Xu, Rui Liu, Kunpeng Liu·· 3 天前AI 评分27
LEAF：用上下文记忆与自蒸馏提升 LLM 推理的探索效率
Explore with Memory, Learn without It: In-Context Exploration and Self-Distillation for LLM Reasoning
AI 导读
LEAF 通过在每次 rollout 前向上下文注入动态的问题专属记忆（含最近一次回答、奖励及环境反馈），并在每次验证后更新，从而将策略更新与 rollout 解耦、以自蒸馏方式更新策略。在四个 benchmark 和两个开源模型上，其探索速度约为随机采样基线的 4 倍，在未见测试集上准确率较最强基线分别提升 1.59 和 1.83 个百分点，且算力消耗低于所有 RL 基线。
正文 · 原文
Abstract:Exploration bounds learning efficiency in reinforcement learning with verifiable rewards (RLVR). Current methods explore mostly through independent sampling, without fully using the information gathered along the way, such as known wrong answers and environment feedback. We propose LEAF, which injects a dynamic, problem-specific memory into the context before every rollout. The memory contains the most recent response, its reward, and other environment feedback. It is updated after every verifier call. We decouple policy updates from rollouts and update the policy through self-distillation. On four benchmarks and two open models, LEAF accelerates exploration by about 4x compared with a random sampling baseline. On unseen test sets, it improves accuracy over the strongest baseline by 1.59 and 1.83 percentage points on the two models, while using less compute than any RL baseline. These results highlight the potential of LEAF in improving both exploration and learning efficiency. Code is available at: this https URL
| Comments: | preprint |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.17734 [cs.LG] |
| (or arXiv:2610.17734v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17734 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yutong Chen [view email]
[v1]
Tue, 13 Oct 2026 05:13:10 UTC (281 KB)
来源：arXiv cs.AI · arxiv.org