arXiv cs.AI· Changyeon Kim, Kimin Lee, Yashraj Narang, Yijie Guo·· 3 天前AI 评分38
ACCRUE：面向机器人基础模型的持续离线策略强化学习框架
ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models
AI 导读
ACCRUE 是一个持续离线策略强化学习框架，通过在任务间传递轻量 actor 和 critic 网络，让每个阶段复用先前 RL 获得的行为与价值知识。在仿真和真机接触密集型装配任务中，ACCRUE 加速后续任务学习且几乎无遗忘，真机训练时间减少 31.0% 至 68.5%。
正文 · 原文
Abstract:Reinforcement learning (RL) post-training improves robot foundation models (RFMs) beyond their demonstrations, but real-world deployment demands more: learning new tasks efficiently while retaining previously acquired behaviors. Existing continual RL methods on top of VLAs mitigate forgetting but rely on on-policy updates with limited forward transfer, whereas off-policy methods improve single-task sample efficiency without addressing learner persistence across tasks. We propose ACCRUE, a continual off-policy RL framework that carries lightweight actor and critic networks across tasks, allowing each stage to reuse behavioral and value knowledge acquired through prior RL. The inherited critic adapts only on current-task transitions and guides current-task actor learning, while prior behavior is preserved by distilling the actor toward cached targets on a small number of retained episodes. Fixed visual features keep the inherited learner and its replay compatible with base-policy updates. Across contact-rich assembly sequences in simulation and on real-robot hardware, ACCRUE accelerates subsequent-task learning with near-zero forgetting and significantly reduces real-robot training time by 31.0\% to 68.5\%.
| Comments: | Project website: this https URL |
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.18596 [cs.RO] |
| (or arXiv:2610.18596v1 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2610.18596 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Changyeon Kim [view email]
[v1]
Tue, 13 Oct 2026 15:53:26 UTC (4,284 KB)
来源：arXiv cs.AI · arxiv.org