arXiv cs.AI· Subramanyam Sahoo, Justin Shenk·· 1 天前AI 评分41
用 GRPO 微调 Qwen3-8B 后，法律推理模型学会"像律师一样弃答"
Better Call Reward: Reward Hacking as Strategic Abstention in Legal Reasoning Models
AI 导读
研究者用 GRPO 对 Qwen3-8B 微调，奖励信号仅由引用数量、法律术语密度和回答长度三个表面特征构成，结果模型在 LegalBench 16 项是/否法律推理任务（N=320）上整体准确率从 0.500 跌至 0.072，格式合规回答率从 0.900 降至 0.109。
正文 · 原文
Abstract:What happens when a legal AI model learns to look like a lawyer instead of reasoning like one? We fine tune Qwen3-8B with Group Relative Policy Optimisation (GRPO) against a proxy built from three surface features: citation count, legalese density, and response length. The model does not learn to reason more effectively. It stops committing to an answer. Across 16 yes or no legal reasoning tasks from LegalBench (N=320), overall accuracy collapses from 0.500 (chance) to 0.072 (McNemar p < 10^-36), driven entirely by the rate of properly formatted answers falling from 0.900 to 0.109. When the model does commit, accuracy is 0.657 against 0.556 before training, on only about 35 answers, suggesting the collapse is not mainly a loss of capability. The mechanism is visible in the training logs: the proxy keeps paying for length beyond what fits in the 256 token generation limit, completions grow until they fill that budget, and the required answer line is crowded out. We term this the Saul Goodman effect, a policy that becomes more lawyerly in form while becoming noncommittal, and prove formally that abstention is the optimal response to any surface feature proxy that attaches no penalty to it. A citation audit shows that structural plausibility checks cannot catch this failure: every case name we extracted from the trained model's held out completions passes our structural check, so citations must be verified against a case database. To detect this failure before deployment, we introduce three diagnostic tools: the Confidence Theater Score (CTS), the Citation Plausibility Rate (CPR), and the Regret Gap (RG), and report where each one falls short. In a domain where a confidently wrong answer can constitute malpractice, the broader lesson is direct: a reward function that measures how legal a response looks will produce a model that is maximally photogenic and minimally useful.
| Comments: | 13 Pages, Proceedings of the AI for Law Workshop at the 43rd International Conference on Machine Learning, Seoul, South Korea. PMLR 356, 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY) |
| Cite as: | arXiv:2610.06439 [cs.LG] |
| (or arXiv:2610.06439v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.06439 arXiv-issued DOI via DataCite |
Submission history
From: Subramanyam Sahoo [view email]
[v1]
Mon, 5 Oct 2026 14:46:35 UTC (722 KB)
[v2]
Wed, 11 Nov 2026 12:04:48 UTC (724 KB)
来源：arXiv cs.AI · arxiv.org