arXiv cs.CL· Guowei Xu, Mert Yuksekgonul, James Zou·· 11 小时前AI 评分38
大语言模型中的稀疏奖励子系统
Sparse Reward Subsystem in Large Language Models
AI 导读
研究发现 LLM 隐藏状态中的奖励相关信息集中在稀疏的神经元子集上，通过简单探针可识别出两类神经元：激活值预测状态价值的 value neurons，以及编码步级时序差分（TD）误差的 dopamine neurons，二者共同构成 LLM 隐藏状态中的稀疏奖励子系统。
正文 · 原文
Abstract:Recent studies show that LLM hidden states encode reward-related information, such as answer correctness and model confidence. However, existing approaches typically fit black-box probes on the full hidden states, offering little insight into how this information is structured across neurons. In this paper, we show that reward-related information is concentrated in a sparse subset of neurons. Using simple probing, we identify two types of neurons: value neurons, whose activations predict state value, and dopamine neurons, whose activations encode step-level temporal difference (TD) errors. Together, these neurons form a sparse reward subsystem within LLM hidden states. These names are drawn by analogy with neuroscience, where value neurons and dopamine neurons in the biological reward subsystem also encode value and reward prediction errors, respectively. We demonstrate that value neurons are robust and transferable across diverse datasets and models, and provide causal evidence that they encode reward-related information. Finally, we show applications of the reward subsystem: value neurons serve as effective predictors of model confidence, and dopamine neurons can function as a process reward model (PRM) to guide inference-time search. Our code is available at this https URL.
| Comments: | NeurIPS 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2602.00986 [cs.CL] |
| (or arXiv:2602.00986v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2602.00986 arXiv-issued DOI via DataCite |
Submission history
From: Guowei Xu [view email]
[v1]
Sun, 1 Feb 2026 02:55:31 UTC (1,397 KB)
[v2]
Mon, 11 May 2026 05:42:58 UTC (1,405 KB)
[v3]
Wed, 4 Nov 2026 14:15:27 UTC (1,483 KB)
来源：arXiv cs.CL · arxiv.org