arXiv cs.AI· Ashish Bansal, Hossam Amer, Keerat Kaur Guliani, Parinita Edke, Lesley Matheson, Leizer Teran·· 12 小时前AI 评分31
专家引导的评分标准作为非可验证领域强化学习的密集奖励信号
Expert-Guided Rubrics as Dense Reward Signals for Reinforcement Learning in Non-Verifiable Domains
AI 导读
研究提出将少量固定的专家判断转化为密集奖励，并与检索指标结合，构建兼容 GRPO 的混合训练信号，用于金融服务等非可验证领域的 LLM 智能体训练。实验显示，相比稀疏奖励基线，Recall@3 提升最高 6%，MRR@3 提升 6.7%。该方法针对稀疏奖励下组内采样结果一致导致 GRPO 优化停滞的问题。
正文 · 原文
Abstract:Large language models (LLMs) are increasingly deployed as autonomous agents in non-verifiable domains such as financial services, where they must interpret user intent, rewrite queries, and retrieve information under domain-specific constraints. Although reinforcement learning (RL) can train such agents from the consequences of their actions, recent successes largely rely on verifiable tasks with automatically computed binary rewards. In non-verifiable retrieval tasks, assessing query quality requires scarce subject-matter expert (SME) judgment that is inherently graded rather than binary. With Group Relative Policy Optimization (GRPO), sparse rewards yield no useful learning signal when all sampled outputs within a group receive the same outcome, causing optimization to stall. We introduce a method that converts a small, fixed investment of SME judgment into a dense reward and combines it with retrieval-based metrics to form a hybrid, GRPO-compatible training signal. Experiments show improvements of up to 6\% in Recall@3 and 6.7\% in MRR@3 compared with sparse-reward baselines.
| Comments: | 11 pages, 4 figures, RL4LLM-Agents workshop ICAIF ACM |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR) |
| Cite as: | arXiv:2610.26932 [cs.LG] |
| (or arXiv:2610.26932v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.26932 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ashish Bansal [view email]
[v1]
Sat, 17 Oct 2026 19:01:11 UTC (625 KB)
来源：arXiv cs.AI · arxiv.org