arXiv cs.CL· Sitao Cheng, Tianle Li, Xuhan Huang, Xunjian Yin, Difan Zou·· 3 天前AI 评分29
DERL：双演化强化学习实现自动化奖励设计
Dual-Evolving Reinforcement Learning for Automated Reward Design
AI 导读
研究提出双演化强化学习框架 DERL，通过外层提议器与内层策略的双重训练实现奖励结构的自主发现，提议器利用内层策略的验证表现进行策略梯度学习。在 ALFWorld、ScienceWorld、GSM8K 和 MATH 等推理任务上，DERL 优于强基线，在分布外任务上提升显著，并在数学任务中改进了结果奖励。代码与模型已开源。
正文 · 原文
Abstract:Crafting effective reward signals remains a central challenge in reinforcement learning (RL) for complex reasoning. Existing methods for automated reward optimization search with a fixed proposer which never learns from its costly trials. We present Dual-Evolving Reinforcement Learning (DERL), a dual-training framework for autonomous discovery of reward structures. An outer-loop proposer composes a reward from task primitives, and an inner policy trains with this reward. The proposer itself learns by policy gradient from the validation performance of the inner policy. Thus, DERL utilizes past-trial feedback to shape its reward-proposal distribution, an empirical meta-gradient of task success. We validate DERL on embodied (ALFWorld), scientific (ScienceWorld), and mathematical (GSM8K, MATH) reasoning. On multi-turn agent benchmarks, DERL outperforms strong baselines, significantly on out-of-distribution tasks. In mathematics, DERL improves on the outcome reward where directly added primitives hurt. Further analyses under matched compute show that our proposer effectively captures the meta-gradient to discover useful reward signals.
| Comments: | Work in Progress. We release our code and model at this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2512.13399 [cs.AI] |
| (or arXiv:2512.13399v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2512.13399 arXiv-issued DOI via DataCite |
Submission history
From: Sitao Cheng [view email]
[v1]
Mon, 15 Dec 2025 14:50:08 UTC (583 KB)
[v2]
Wed, 13 May 2026 04:43:51 UTC (646 KB)
[v3]
Tue, 13 Oct 2026 21:12:48 UTC (1,275 KB)
来源：arXiv cs.CL · arxiv.org