arXiv cs.AI· Yuan Zhuang, Yuexin Bian, Sihong He, Jie Feng, Qing Su, Songyang Han, Jonathan Petit, Shihao Ji, Yuanyuan Shi, Fei Miao·· 14 小时前AI 评分27
用LoRA做结构正则：离策略强化学习中的Critic学习新方法
Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learning
AI 导读
研究者提出用LoRA作为critic学习的结构正则化手段：冻结随机初始化的基础矩阵，只优化低秩适配器，将权重变化约束为低秩残差。在SAC+SimbaV2、BRC（DMC-Hard）与FastTD3及其SimbaV2变体（IsaacLab）上评测，LoRA降低了训练中的critic损失，多数任务取得最佳或具竞争力表现。该工作入选NeurIPS 2026。
正文 · 原文
Abstract:Scaling critic capacity is a promising direction for improving off-policy reinforcement learning (RL). However, recent work shows that larger critics are prone to overfitting and instability in replay-based bootstrapped training. In this paper, we propose using Low-Rank Adaptation (LoRA) as a structural regularizer for critic learning from scratch. Our approach freezes randomly initialized base matrices and optimizes only the corresponding low-rank adapters, thereby constraining the learned weight changes to low-rank residuals. We evaluate our method on state-based continuous-control tasks, using SAC with SimbaV2 and BRC on DMC-Hard, and FastTD3 with its SimbaV2 variant on IsaacLab. Empirically, LoRA reduces critic loss during training and improves overall policy performance, achieving the best or competitive results on most tasks. Extensive experiments demonstrate that our low-rank updates provide a simple and effective form of structural regularization for critic learning in off-policy RL.
| Comments: | 2026 Conference on Neural Information Processing Systems (NeurIPS 2026) |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2604.18978 [cs.LG] |
| (or arXiv:2604.18978v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.18978 arXiv-issued DOI via DataCite |
Submission history
From: Yuan Zhuang [view email]
[v1]
Tue, 21 Apr 2026 01:59:54 UTC (4,126 KB)
[v2]
Thu, 7 May 2026 15:47:31 UTC (2,855 KB)
[v3]
Thu, 29 Oct 2026 16:01:54 UTC (2,888 KB)
来源：arXiv cs.AI · arxiv.org