arXiv cs.CL· Gaotang Li, Bhavana Dalvi Mishra, Zifeng Wang, Jun Yan, Yanfei Chen, Chun-Liang Li, Long T. Le, Rujun Han, George Lee, Hanghang Tong, Chen-Yu Lee, Tomas Pfister·· 12 小时前AI 评分35
RubricEM：用评分标准引导策略分解的元强化学习框架，超越可验证奖励
RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards
AI 导读
RubricEM 是一个面向深度研究智能体的统一强化学习框架，结合分阶段信用分配与基于反思的元强化学习实现经验复用。该框架先用监督微调让智能体学会用评分标准规划并将轨迹组织为语义阶段，再以 Stage-Structured GRPO 优化任务策略，同时异步训练共享骨干的反思元策略。RubricEM-8B 在四个长文研究基准上超越更大规模的开源模型，接近专有深度研究系统。
正文 · 原文
Authors:Gaotang Li, Bhavana Dalvi Mishra, Zifeng Wang, Jun Yan, Yanfei Chen, Chun-Liang Li, Long T. Le, Rujun Han, George Lee, Hanghang Tong, Chen-Yu Lee, Tomas Pfister
Abstract:Deep research agents plan, search, evaluate evidence, and synthesize long-form reports. Training these agents with reinforcement learning (RL) is challenging because they operate over long horizons and their outputs lack exact answer verifiers. To this end, we introduce RubricEM, a unified RL framework for deep research that combines stagewise credit assignment with reflection-based meta-RL for experience reuse. Rubrics unify the framework by providing a shared interface for policy execution, judge feedback, and experience reuse. RubricEM first instills a rubric-guided scaffold through supervised fine-tuning, teaching the agent to plan with rubrics and organize its trajectory into semantic stages. Building on this structural prior, we optimize the task policy with Stage-Structured GRPO, using stagewise judge feedback for finer-grained credit assignment. In parallel, we asynchronously train a shared-backbone reflection meta-policy (with meta-RL) to distill completed trajectories into reusable guidance, contributing through both shared-parameter updates and textual memory. The resulting RubricEM-8B model achieves strong performance across four long-form research benchmarks, outperforming existing open models of larger sizes and approaching proprietary deep-research systems. Beyond final performance, we perform thorough controlled analyses to understand the key ingredients of RubricEM.
| Comments: | 67 pages, 6 figures |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.10899 [cs.CL] |
| (or arXiv:2605.10899v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.10899 arXiv-issued DOI via DataCite |
Submission history
From: Gaotang Li [view email]
[v1]
Mon, 11 May 2026 17:40:38 UTC (2,225 KB)
[v2]
Fri, 16 Oct 2026 11:35:29 UTC (645 KB)
来源：arXiv cs.CL · arxiv.org