arXiv cs.AI· Jiamin He, Mark Rowland, Zhaohan Daniel Guo, Hado van Hasselt, Diana Borsa·· 10 小时前AI 评分22
用 Excursion 策略实现 Off-policy 学习
Off-policy Learning with Excursion Policies
AI 导读
研究提出 excursion 策略：智能体先按目标策略执行多步“excursion”，再切回行为策略，作为行为策略与目标策略线性插值之外的一种结构化折中。作者给出该策略的动态规划结果，并推导出可证明正确的 off-policy TD 算法用于估计其价值。在连续控制基准上，其效率-稳定性权衡与 mixture 策略相当，但行为机制存在概念差异。
正文 · 原文
Abstract:Off-policy reinforcement learning (RL) tries to learn the value of a target policy while data comes from a different behavior policy. This often requires trading off bias for variance or stability. One way to trade these off is to linearly interpolate between the behavior and target policies, but this represents only one point in a broader space of interpolation mechanisms. We introduce excursion policies--a structured alternative where the agent follows the target policy for a multi-step "excursion" before reverting to the behavior policy. We derive dynamic programming results for these policies, establishing the theoretical basis for their use in RL. We further develop provably sound off-policy temporal-difference (TD) algorithms for excursion value estimation. We find that excursion policies provide an efficiency-stability trade-off comparable to mixture policies in continuous control benchmarks, and our analysis illustrates conceptual differences in their behavior.
| Comments: | Accepted to NeurIPS 2026 (camera-ready) |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.13275 [cs.LG] |
| (or arXiv:2611.13275v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13275 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiamin He [view email]
[v1]
Mon, 9 Nov 2026 21:41:23 UTC (1,742 KB)
来源：arXiv cs.AI · arxiv.org