arXiv cs.AI· Zhengqing Yan, Fan Guo, Xinyang Liu, Yi Zhang, Yao Liu, Qi Liu, Jihao Huang, Kang Song·· 3 天前AI 评分20
GDEPO：面向样本高效强化学习的组双动态与等权优势策略优化
GDEPO: Group Dual-dynamic and Equal-right Advantage Policy Optimization for Sample-Efficient Reinforcement Learning
AI 导读
针对 GRPO 在自动定理证明中相对优势估计与二元验证反馈冲突、静态采样丢弃无效批次造成数据浪费的问题，研究者提出 GDEPO，通过动态补充采样、等权优势（解耦优势函数的符号与幅度）和动态额外迭代三项机制提升数据利用率。在 MinF2F-test、MathOlympiadBench 和 PutnamBench 上的实验验证了其有效性，消融研究确认各组件协同作用。
正文 · 原文
Abstract:Automated Theorem Proving (ATP) is a fundamental AI challenge, requiring machine-verifiable proofs in formal languages like Lean to evaluate reasoning. Reinforcement Learning (RL), especially Group Relative Policy Optimization (GRPO), is a mainstream approach. However, GRPO faces two critical issues in ATP: its relative advantage estimation may conflict with binary verifier feedback under composite rewards, and its static sampling strategy discards batches without valid proofs, causing data waste. Repeated training on mastered problems also wastes computation. To address this, we propose Group Dual-dynamic and Equal-right Advantage Policy Optimization (GDEPO), incorporating three core mechanisms: 1) dynamic additional sampling, resampling ineffective batches until a valid proof is found; 2) equal-right advantage, decoupling the advantage function's sign (correctness) from its magnitude (auxiliary rewards) to ensure stable policy updates while increasing effective computation; and 3) dynamic additional iterations, applying extra gradient steps to initially failed but eventually successful samples to accelerate learning. Experiments on MinF2F-test, MathOlympiadBench, and PutnamBench confirm GDEPO's effectiveness, with ablation studies validating its synergistic components. The method enhances data utilization and optimization efficiency, offering a novel training paradigm for ATP.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2601.06795 [cs.AI] |
| (or arXiv:2601.06795v4 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2601.06795 arXiv-issued DOI via DataCite |
Submission history
From: Zhengqing Yan [view email]
[v1]
Sun, 11 Jan 2026 07:34:41 UTC (2,146 KB)
[v2]
Wed, 21 Jan 2026 02:58:47 UTC (2,146 KB)
[v3]
Thu, 22 Jan 2026 14:58:18 UTC (2,146 KB)
[v4]
Mon, 12 Oct 2026 13:47:23 UTC (2,103 KB)
来源：arXiv cs.AI · arxiv.org