跳到正文
arXiv cs.AI· Jan Kirin·· 10 小时前AI 评分29

从循环 Transformer 内部状态读取人类偏好

Reading Human Preference from Looped Transformer Internal States

AI 导读

在每 token 运行四遍层的 Ouro-2.6B-Thinking 上,用 Anthropic HH-RLHF 数据可从隐藏状态判断人类更偏好哪条回复:线性探针准确率 61.5%,分别打分的非线性小网络 64.9%,v1 评估器在测试集达 64.7%。

正文 · 原文

View PDF HTML (experimental)

Abstract:Show a frozen language model two replies to the same request, one of which a person preferred. Can a small probe on the model's hidden states tell which? On Ouro-2.6B-Thinking, a model that runs its layers four times per token, and Anthropic's HH-RLHF data, the answer is yes, but only faintly. A linear probe picks the preferred reply 61.5% of the time, a small nonlinear network that scores each reply separately 64.9%, and the evaluator trained for version 1 reaches 64.7% on the test split: clearly better than simple rules such as picking the longer reply, but weak. Whether preference is stored as a comparison between the two replies or as a property of each is still open. A linear probe cannot tell, and a network that reads both replies together has an estimated edge of only 0.7 points (95% CI 0.0 to 1.4) over one that scores each reply separately. These numbers hold under three controls: scoring every pair in both orders, splitting the data by pair, and, for the probes, keeping the end of each conversation, where the two replies differ. Version 1's 95.2% and 84.5% were obtained without the first two and do not hold; version 2's corrected relational-probe accuracy, obtained without the third, was four to five points too low. This version replaces both.
Comments: Version 3 replaces the original manuscript and v2 erratum with a rewritten analysis. It corrects the probe truncation that could remove the replies being compared, tests the original evaluator in both reply orders, and compares linear and nonlinear probes that score replies separately or jointly on pair-disjoint splits
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2604.09870 [cs.LG]
  (or arXiv:2604.09870v3 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2604.09870

arXiv-issued DOI via DataCite

Submission history

From: Jan Kirin [view email]
[v1] Fri, 10 Apr 2026 20:00:49 UTC (528 KB)
[v2] Thu, 16 Jul 2026 14:16:06 UTC (533 KB)
[v3] Wed, 18 Nov 2026 13:32:12 UTC (79 KB)

来源:arXiv cs.AI · arxiv.org