arXiv cs.AI· Sami Nur Islam, Johan Obando-Ceron, Doina Precup, Aaron Courville, Pablo Samuel Castro·· 13 小时前AI 评分31
冻结而非克隆：不保留离线数据时用表达性策略微调
Freeze, Don't Clone: Fine-Tuning with Expressive Policies Without Retaining Offline Data
AI 导读
研究指出，离线数据不可用时继续在线行为克隆会迅速抹掉预训练先验，导致表征坍缩与动作多样性丧失，使 critic 缺乏有效探索。在七个 OGBench 域上，停止在线行为拟合、冻结预训练表达性先验，可显著改善 IFQL、FQL 与 QC 的无保留微调效果。该失败被归因于行为反馈循环，冻结先验可保留离线覆盖度并让价值学习持续驱动改进。
正文 · 原文
Abstract:Offline-to-online reinforcement learning uses offline pretraining to accelerate subsequent online learning. While many methods assume the offline dataset remains available throughout fine-tuning, there are many settings where this is not the case. Prior work attributes failures in this setting primarily to critic recalibration under distribution shift. We identify a complementary actor-side failure: once the offline data disappears, continued behavioral cloning (BC) no longer preserves the offline behavior distribution and instead refits the behavioral model to the agent's limited, and nonstationary, online experience. We demonstrate that stopping online behavioral fitting while preserving the pretrained expressive prior substantially improves no-retention fine-tuning across implicit flow Q-learning (IFQL), flow Q-learning (FQL), and action-chunked RL (QC) on seven OGBench domains. We trace this failure to a behavioral feedback loop: continuing online behavioral cloning rapidly erases the pretrained prior, inducing representational collapse and action diversity loss that starves the critic of useful exploration. Freezing the prior breaks this cycle, preserving offline coverage while allowing value learning to drive continuous improvement.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.36247 [cs.LG] |
| (or arXiv:2610.36247v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36247 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Johan Obando-Ceron [view email]
[v1]
Mon, 26 Oct 2026 20:28:50 UTC (1,693 KB)
来源：arXiv cs.AI · arxiv.org