arXiv cs.AI· Boyang Xu, Qing Zou, Siqin Yang, Hao Yan·· 13 小时前AI 评分26
Path-Coupled Bellman Flows：用路径耦合流匹配改进分布式强化学习
Path-Coupled Bellman Flows for Distributional Reinforcement Learning
AI 导读
研究者提出 Path-Coupled Bellman Flows（PCBF），一种用流匹配学习回报分布的连续时间分布式强化学习（DRL）方法，通过源一致的 Bellman 耦合路径解决流源边界不匹配与自举高方差问题。
正文 · 原文
Abstract:Distributional reinforcement learning (DRL) models the full return distribution, but existing finite-support or quantile-based methods rely on projections, while recent flow-based approaches can suffer from \emph{boundary mismatch} at the flow source or from \emph{high-variance} bootstrapping when current and successor noises are independent. We propose Path-Coupled Bellman Flows (PCBF), a continuous-time DRL method that learns return distributions with flow matching using \textbf{source-consistent Bellman-coupled paths}: the current path starts from the required base prior at $t{=}0$, reaches the Bellman target at $t{=}1$, and maintains a pathwise affine relation to the successor flow at intermediate times (without requiring time-$t$ marginals to satisfy a distributional Bellman fixed point for all $t$). PCBF couples current and successor return flows through shared base noise and uses a $\lambda$-parameterized control-variate target: $\lambda{=}0$ recovers an unbiased sample Bellman target, while $\lambda{>}0$ trades controlled bias for variance reduction. Experiments on analytically tractable MRPs, OGBench, and D4RL show improved distributional fidelity and training stability, and competitive offline RL performance.
| Comments: | Accepted to the 43rd International Conference on Machine Learning (ICML 2026) |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| ACM classes: | I.2.6 |
| Cite as: | arXiv:2605.08253 [cs.LG] |
| (or arXiv:2605.08253v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.08253 arXiv-issued DOI via DataCite |
|
| Journal reference: | Proceedings of the 43rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026 |
Submission history
From: Hao Yan [view email]
[v1]
Thu, 7 May 2026 19:05:01 UTC (8,030 KB)
[v2]
Wed, 3 Jun 2026 21:28:18 UTC (8,056 KB)
[v3]
Tue, 20 Oct 2026 18:03:55 UTC (8,062 KB)
来源：arXiv cs.AI · arxiv.org