arXiv cs.AI· Tingrui Huang, Ai Bo, Junzhe Zhang·· 3 天前AI 评分20
CauCRL：面向离线对比目标条件强化学习的反事实 rollout 方法
Counterfactual Rollouts for Offline Contrastive Goal-Conditioned Reinforcement Learning
AI 导读
针对离线示范者信息不可见导致对比强化学习（CRL）学到虚假动作—结果关联的问题，研究者提出因果对比强化学习（CauCRL），用混杂马尔可夫决策过程形式化该设定，并引入反事实转移区分数据可识别与不可识别的部分。CauCRL 从正负离线示范中学习悲观的反事实转移，并在反事实 rollout 上训练 critic，在混杂目标达成任务上成功率显著优于标准 CRL。
正文 · 原文
Abstract:Contrastive reinforcement learning (CRL) learns goal-conditioned values by associating state--action pairs with the states subsequently reached, assuming that no unobserved variable jointly affects the recorded action and its outcome. When offline demonstrations come from a demonstrator with information hidden from the learner, this assumption fails and the critic absorbs spurious action--outcome associations. We formalize this setting with confounded Markov decision processes and introduce counterfactual transitions that separate what the data identify from what they do not; minimizing over all compatible transitions yields a worst-case occupancy that lower-bounds the true one. Causal Contrastive Reinforcement Learning (CauCRL) learns a pessimistic counterfactual transition from positive and negative offline demonstrations and trains the critic on counterfactual rollouts, substantially improving success rates over standard CRL on confounded goal-reaching tasks.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14953 [cs.LG] |
| (or arXiv:2610.14953v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14953 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tingrui Huang [view email]
[v1]
Sat, 10 Oct 2026 14:25:32 UTC (648 KB)
来源：arXiv cs.AI · arxiv.org