Value-Rectified Distillation：面向基于流的离线强化学习的行为策略蒸馏框架

View PDF HTML (experimental)

Abstract:We aim to address the problem of Out-of-Distribution (OOD) Mode Averaging, a critical problem that emerges when distilling Flow Matching (FM) based behavior policies in offline Reinforcement Learning (RL): When learning over multi-modal datasets, standard distillation inadvertently forces the policy to interpolate across different behavioral modes, causing the generated actions to collapse into invalid, OOD regions. To resolve this, we propose Value-Rectified Distillation (VRD), a novel framework that reformulates offline policy distillation as a value-guided generative trajectory alignment problem. By employing a dynamic coupling mechanism, VRD actively rectifies flow dynamics to explicitly decouple distinct behavioral modes, cleanly isolating high-value actions. Specifically, we instantiate VRD through two complementary algorithms: VRD-Q, which utilizes value-quantile behavior distillation to statistically filter out low-value behaviors, and VRD-R, which reorganizes the latent space to geometrically isolate high-value generative trajectories from the FM-based behavior policy. Extensive empirical evaluations demonstrate that VRD improves existing generative baselines across diverse continuous control and planning tasks on the D4RL, OGBench, and Minari benchmarks, especially over those multi-modal, non-expert datasets.

Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2610.29075 [cs.LG]
	(or arXiv:2610.29075v1 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2610.29075 arXiv-issued DOI via DataCite (pending registration)
Journal reference:	K. Jiang, W. Jiang, Y. Kawahara, and X. Tan, "Value-Rectified Distillation for Flow-based Offline Reinforcement Learning," Advances in Neural Information Processing Systems 39, 2026

Submission history

From: Ke Jiang [view email]
[v1] Wed, 21 Oct 2026 01:58:00 UTC (4,766 KB)