arXiv cs.MA· Pengcheng Dai, Dongming Wang, Wenwu Yu, Wei Ren·· 2 天前AI 评分18
面向网络化多智能体强化学习的分布式可扩展耦合策略算法 DSCP
Distributed scalable coupled policy algorithm for networked multi-agent reinforcement learning
AI 导读
针对奖励相互依赖、策略相互耦合的网络化多智能体强化学习（NMARL），研究者提出分布式可扩展耦合策略（DSCP）算法，引入"邻居平均 Q 函数"并推导出耦合策略梯度的新表达式。
正文 · 原文
Abstract:This paper studies networked multi-agent reinforcement learning (NMARL) with interdependent rewards and coupled policies. In this setting, each agent's reward depends on its own state-action pair as well as those of its direct neighbors, and each agent's policy is parameterized by its local parameters together with those of its $\kappa_{p}$-hop neighbors, with $\kappa_{p}\geq 1$ denoting the coupled radius. The objective of the agents is to collaboratively optimize their policies to maximize the discounted average cumulative reward. To address the challenge of interdependent policies in collaborative optimization, we introduce a novel concept termed the ``neighbors' averaged $Q$-function'' and derive a new expression for the coupled-policy gradient. Based on these theoretical foundations, we develop a distributed scalable coupled policy (DSCP) algorithm, where each agent relies only on the state-action pairs of its $\kappa_{p}$-hop neighbors and the rewards of its $(\kappa_{p}+1)$-hop neighbors. Specifically, in the DSCP algorithm, we employ a geometric 2-horizon sampling method that does not require storing a full $Q$-table to obtain an unbiased estimate of the coupled policy gradient. Moreover, each agent interacts exclusively with its direct neighbors to obtain accurate policy parameters, while maintaining local estimates of other agents' parameters to execute its local policy and collect samples for optimization. These estimates and policy parameters are updated via a push-sum protocol, enabling distributed coordination of policy updates across the network. We prove that the joint policy produced by the proposed algorithm converges to a first-order stationary point of the objective function. Finally, the effectiveness of DSCP algorithm is demonstrated through simulations in a robot path planning environment, showing clear improvement over state-of-the-art methods.
| Subjects: | Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2512.05447 [cs.MA] |
| (or arXiv:2512.05447v3 [cs.MA] for this version) | |
| https://doi.org/10.48550/arXiv.2512.05447 arXiv-issued DOI via DataCite |
Submission history
From: Pengcheng Dai [view email]
[v1]
Fri, 5 Dec 2025 05:51:14 UTC (1,070 KB)
[v2]
Wed, 10 Dec 2025 06:51:16 UTC (1,071 KB)
[v3]
Thu, 15 Oct 2026 09:12:24 UTC (1,075 KB)
来源：arXiv cs.MA · arxiv.org