Latent Reward Steering：一种在推理 LLM 中隐式促进认知行为的自适应推理时框架

Authors:Jiakang Li, Guanyu Zhu, Can Jin, Chenxi Huang, Dexu Yu, Ronghao Chen, Yang Zhou, Hongwu Peng, Xuanqi Lan, Dimitris N. Metaxas, Youhua Li

View PDF HTML (experimental)

Abstract:Strong reasoning depends not only on model knowledge but also on how effectively cognitive behaviors are deployed during generation. Existing methods often rely on explicit behavior-level control, making them insufficiently adaptive when failures and required corrections vary across reasoning states, tasks, and models. To this end, we propose Latent Reward Steering (LRS), an adaptive inference-time framework that promotes cognitive behaviors by optimizing the sparse-autoencoder (SAE) latent states that implicitly carry them. Rather than relying on predefined cognitive behaviors or steering directions derived from them, LRS trains a latent reward model on reasoning traces by final answer correctness to estimate the quality of intermediate latent states. During inference, reward gradients provide state-specific correction directions for fragile latent states, while a reward and confidence gate restricts intervention to states the reward signal flags as fragile. Experiments on multiple reasoning LLM backbones and benchmarks show that \ours consistently improves performance over various baselines, and post-hoc analyses further indicate that \ours implicitly promotes good cognitive behaviors that fix the original reasoning errors. Code is available at: this https URL.

Submission history

From: Jiakang Li [view email]
[v1] Sat, 30 May 2026 13:38:06 UTC (2,098 KB)
[v2] Fri, 10 Jul 2026 01:15:34 UTC (2,098 KB)
[v3] Fri, 21 Aug 2026 20:44:57 UTC (2,089 KB)
[v4] Thu, 8 Oct 2026 20:53:09 UTC (2,090 KB)