跳到正文
arXiv cs.CL· Liang Yue, Eric A. Vance·· 10 小时前AI 评分52

研究:LLM 评判者的知识难以迁移,但类别偏好会传给 RL 模型

Blind Spots Travel, Knowledge Stays Home: What RL Inherits from an LLM Judge

AI 导读

一项 814 次 RL 实验研究考察 7-8B 模型从 LLM 评判者那里继承了什么,发现评判者对单个答案的知识几乎不迁移,用答案键或更强评判者训练,模型选择新问题作答的能力也不比未训练模型好,评判者知识每百题仅多纠正约一个答案。

正文 · 原文

View PDF HTML (experimental)

Abstract:LLM judges serve as rewards where no answer key exists, in the hope that their knowledge passes to the trained model. In 814 RL runs that teach 7-8B models to answer closed-book questions or abstain, we ask what passes. What a grader knows about single answers stays home. A wrong answer that a judge rewards is memorised for that one question. A model trained with the answer key, or with a far stronger judge, chooses which new questions to answer no better than the untrained model does by sampling 32 answers and abstaining where they disagree; the grader's knowledge adds about one corrected answer per hundred questions. What a grader believes about a kind of question travels. A judge that rejects eleven of the twelve correct training answers about capitals of former states teaches the model to refuse 98% of such capitals on new questions, a loss that accuracy and abstention rates do not reveal. Changing only those twelve verdicts by hand moves the refusal from 5% to 90%, even for a grader that accepts as many answers as the answer key; at that leniency, two of four judges already cost 38-42% of these capitals. The capitals are not forgotten: with "I don't know" blocked, the model answers 94% of them correctly. The grader therefore needs no knowledge of its own: rewarding the model for giving the answer that most of its own untrained samples agree on, with no labels and no judge, earns the answer key's reward and keeps the category.
Comments: 31 pages, 6 figures, 27 tables. Code and data: this https URL
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as: arXiv:2611.25191 [cs.CL]
  (or arXiv:2611.25191v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.25191

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Liang Yue [view email]
[v1] Wed, 18 Nov 2026 21:13:28 UTC (115 KB)

来源:arXiv cs.CL · arxiv.org