arXiv cs.AI· Carl Qi, Weijun Dong, Xiaojie Wang, Peiyi Chen, Rahul Sajnani, Mingyuan Zhang, Amy Zhang, Jose Barreiros, Manikatan Nambi, Lillian J. Ratliff, Dylan Randle·· 12 小时前AI 评分34
AnchorQ：用流反转动作锚点实现样本高效的强化学习策略改进
Sample-Efficient Reinforcement Learning for Policy Improvement via Flow-Reversed Action Anchors
AI 导读
AnchorQ 通过流反转将参考行为映射为状态条件化的隐式锚点，并在锚点周围采样局部粒子、用价值引导采样探索更优动作，实现由粗到细的策略改进。在相同交互预算下，该方法在仿真与真实机器人操作任务中均优于现有后训练基线，真实世界成功率提升 37%，仿真任务吞吐量提升 97%。
正文 · 原文
Authors:Carl Qi, Weijun Dong, Xiaojie Wang, Peiyi Chen, Rahul Sajnani, Mingyuan Zhang, Amy Zhang, Jose Barreiros, Manikatan Nambi, Lillian J. Ratliff, Dylan Randle
Abstract:Generalist robot policies offer strong behavioral priors, but achieving deployment-level reliability and throughput remains difficult. Reinforcement learning (RL) can improve these policies, yet its effectiveness is constrained by the cost of online interaction. Existing methods adapt frozen flow-matching policies by treating noise space as a continuous action space. We show that this behaviorally unstructured space makes exploration inefficient, often preventing these methods from extracting meaningful learning signals on challenging tasks. We introduce AnchorQ, which enables coarse-to-fine policy improvement through a structured, refinable latent action representation. At each decision step, AnchorQ uses flow reversal to map reference behaviors to state-conditioned latent anchors, samples local particles around those anchors, and applies value-guided sampling to explore promising refinements. Our analysis shows that anchors reduce identification complexity at the cost of support bias; particle refinement trades this bias against action-class complexity; and critic accuracy determines the reliability of KL-regularized, value-guided selection. Across simulated and real-world manipulation tasks, AnchorQ outperforms state-of-the-art post-training baselines in sample efficiency, delivering a 37% improvement in real-world success rate and 97% higher simulated task throughput under the same interaction budget. Additional visualizations are available on our website this https URL.
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.22852 [cs.RO] |
| (or arXiv:2610.22852v1 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22852 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Carl Qi [view email]
[v1]
Fri, 16 Oct 2026 02:51:48 UTC (7,152 KB)
来源：arXiv cs.AI · arxiv.org