arXiv cs.AI· Tianyang Luo, Tao Feng, Zhigang Hua, Yan Xie, Shuang Yang, Ge Liu, Jiaxuan You·· 11 小时前AI 评分34
MemReward：用图结构经验记忆为 LLM 奖励预测补全有限标签
MemReward: Graph-Based Experience Memory for LLM Reward Prediction with Limited Labels
AI 导读
MemReward 用异构图把查询与初始策略的 rollout（思考过程与最终答案）连成节点，先在带标签 rollout 上预热 GNN，再在在线 RL 微调中按查询相似度挂载无标签 rollout 并传播奖励。
正文 · 原文
Abstract:Reinforcement learning (RL) fine-tuning improves large language model (LLM) reasoning using reward signals on sampled rollouts. When reward labels are available for only a subset of rollouts, restricting policy updates to that subset leaves the remaining rollouts unused for policy optimization. Semi-supervised learning propagates labels from labeled to unlabeled samples, but existing methods either diffuse labels with fixed similarity weights or treat each sample independently, ignoring the relations among queries, thinking processes, and answers. We propose MemReward, a graph-based experience memory that propagates rewards to unlabeled rollouts during online policy optimization. MemReward stores queries and the rollouts (thinking processes and final answers) of an initial LLM policy as nodes of a heterogeneous graph connected by similarity and structural edges, and warms up a GNN on labeled rollouts to propagate rewards over this graph. During online RL fine-tuning, unlabeled rollouts are attached to the graph by query similarity, and the policy is updated with ground-truth rewards on labeled rollouts and GNN-predicted rewards on the rest. We evaluate MemReward on 13 benchmarks in mathematics, question answering, and code generation with verifiable rewards, and further with LLM-judge, human-preference, and continuous rewards. With ground-truth rewards on only 20% of rollouts, MemReward achieves 96.6% of the performance of fully supervised training (Oracle) on Qwen2.5-1.5B and 97.3% on Qwen2.5-3B, where it also outperforms label propagation and FixMatch-style self-training. It carries over to Llama backbones up to 8B and closely approaches Oracle on out-of-domain tasks.
| Comments: | Accepted to NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.19310 [cs.LG] |
| (or arXiv:2603.19310v5 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2603.19310 arXiv-issued DOI via DataCite |
Submission history
From: Tianyang Luo [view email]
[v1]
Fri, 13 Mar 2026 18:00:16 UTC (426 KB)
[v2]
Tue, 24 Mar 2026 05:09:14 UTC (370 KB)
[v3]
Thu, 21 May 2026 02:26:31 UTC (339 KB)
[v4]
Fri, 22 May 2026 03:50:49 UTC (325 KB)
[v5]
Wed, 4 Nov 2026 05:07:38 UTC (347 KB)
来源：arXiv cs.AI · arxiv.org