TRQAM：基于信任域 Q Adjoint Matching 的稳定离线强化学习微调

View PDF HTML (experimental)

Abstract:Off-policy reinforcement learning of pretrained flow policies remains challenging due to the instability of optimization arising from the multi-step sampling process. Recently, Q-learning with Adjoint Matching (QAM) addressed this by recasting policy improvement as a stochastic optimal control (SOC) problem guided by a learned critic. However, QAM inherits a fundamental fragility of critic-guided improvement, since small critic errors can be exponentially amplified and often lead to performance collapse. This paper introduces Trust Region Q Adjoint Matching (TRQAM), a stable off-policy fine-tuning algorithm that adaptively controls the path-space KL between the fine-tuned and pretrained policies through projected dual descent. Specifically, we adapt a trust-region parameter $\lambda$ inside the sampling process of the flow policy and prove that $\lambda$ exactly weights the path-space KL in the SOC objective. As a result, our method can tightly control the deviation from the pretrained policy within a prescribed KL budget, achieving stable off-policy RL. Through experiments on 50 OGBench tasks, TRQAM consistently outperforms prior methods in both offline RL and offline-to-online RL. In particular, TRQAM achieves an overall success rate of 68% in offline RL, substantially improving over the strongest baseline at 46%.

Comments:	NeurIPS 2026. Code: this https URL, Blog: this https URL
Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO)
Cite as:	arXiv:2605.27079 [cs.LG]
	(or arXiv:2605.27079v2 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2605.27079 arXiv-issued DOI via DataCite

Submission history

From: Yonghoon Dong [view email]
[v1] Tue, 26 May 2026 14:28:43 UTC (1,193 KB)
[v2] Thu, 8 Oct 2026 08:52:10 UTC (1,205 KB)