arXiv cs.CL· Hyun Ryu, Jong-Heon Kim, Yiqing Xie, Seungone Kim, Carolyn Rose, Eunho Yang·· 12 小时前AI 评分34
FORK 奖励：用精度目标缓解 rubric-based RL 中的奖励黑客问题
Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL
AI 导读
FORK 奖励通过联合优化召回率与新增的精度目标，缓解 rubric-based RL 中模型为追求覆盖率而变得冗长、混入错误内容的问题。精度目标要求回答的每个单元既必要又事实正确，分别由 rubric 驱动的必要性评分和事实正确性评分器衡量。
正文
Abstract:Rubric-based rewards enable effective reinforcement learning on non-verifiable tasks. However, because these rewards are typically defined as a weighted measure of rubric coverage, models are trained primarily to maximize a recall objective. As a result, RL-trained models produce responses that are more complete but also more verbose and contain more unnecessary or factually incorrect content, leading them to be less preferred than responses from the base model. To address this issue, we present the FORK reward, which jointly optimizes recall and a novel precision objective, which requires each unit of a model response to be both necessary and factually correct, as measured by rubric-driven necessity credits and a factual-correctness grader, respectively. Our experimental results show that FORK outperforms existing rubric-based RL methods and achieves Pareto-optimal performance that balances rubric coverage and overall response quality. Compared to the original recall-based reward, FORK substantially improves precision and reduces response length while achieving similar rubric coverage, which leads to significantly higher win rate compared to the base policy (e.g., from 10.6% to 68.0% on RubricHub-Medical and from 22.1% to 74.1% on HealthBench).
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.33512 [cs.CL] |
| (or arXiv:2610.33512v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33512 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hyun Ryu [view email]
[v1]
Sat, 24 Oct 2026 07:43:06 UTC (1,479 KB)
来源：arXiv cs.CL · arxiv.org