arXiv cs.MA· Wei Dai, Qianchu Liu, Guanghui Qin, Xing Han, Tristan Naumann, Paul Pu Liang, Sheng Zhang·· 2 天前AI 评分37
RRI：用对抗式元提示演化实现递归奖励改进
Recursive Reward Improvement via Adversarial Meta-Prompt Evolution
AI 导读
针对开放式专家任务中模型自写评分标准易被 solver 钻空子的问题，研究者提出 RRI（Recursive Reward Improvement），用奖励黑客对抗者监督奖励：先让对抗者写出钻评分标准空子的无用答案，再让同一模型在不看评分标准的情况下写诚实答案，经最多两步修复使诚实答案胜出。
正文 · 原文
Abstract:Recursive Self-Improvement (RSI) has gained wide attention for its ability to work with limited data. In a proposer-solver loop, a proposer model writes training tasks and a solver model learns to solve them through reinforcement learning. Such loops have succeeded in mathematics and coding, where every answer can be checked automatically. Open-ended expert tasks, like clinical report generations, have no answer key, which makes simple question-answer generation not viable. Here the proposer must also write a rubric that lists the content an answer should include and the errors it should avoid. However, the unique challenge behind model-written rubric is the comprehensiveness of it. For naively generated rubric, solvers tend to exploit it, naming the expected topics without the substance behind them, which corrupts the learning process. To solve this problem, we introduce RRI, Recursive Reward Improvement, which uses a reward hacking adversary to supervise the reward. Before the solver trains on a generated task, an adversary writes an answer that games its rubric while being as useless as possible, the same model writes an honest answer without seeing the rubric, and up to two repair steps revise the rubric to make the honest answer win. The exploits found also rewrite how future tasks and rubrics are generated. On HealthBench Professional, RRI improves the adjusted score over the Generation Only self-improvement baseline by 10.9 points at 9B and 12.4 at 27B, and by 10.3 points without any external model. In particular, the gains are largest on difficult and writing tasks. We show that the 27B model trained on the HealthBench Professional description improves by 6.6 points on PRBench Hard and 8.0 on ProfBench without seeing a task from either. We further analyze the exploitable points of model-generated question-rubric pairs to facilitate future research on open-ended self-play.
| Subjects: | Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA) |
| ACM classes: | I.2.6; I.2.7; J.3 |
| Cite as: | arXiv:2610.21040 [cs.AI] |
| (or arXiv:2610.21040v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21040 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Wei Dai [view email]
[v1]
Thu, 15 Oct 2026 00:31:18 UTC (260 KB)
来源：arXiv cs.MA · arxiv.org