arXiv cs.MA· Seongmin Kim, Woohyeon Byeon, Jiwon Jeon, Seungyul Han, Youngchul Sung·· 9 小时前AI 评分24
LEICA:应对合作多智能体强化学习中的外生变异性
Addressing Exogenous Variability in Cooperative Multi-Agent Reinforcement Learning
AI 导读
针对对手策略或环境机制变化等外生变异性导致合作多智能体强化学习失效的问题,研究者将其形式化为 Exogenous Dec-POMDP(ED-POMDP),并提出 CTDE 兼容算法 LEICA,通过历史条件上下文表示与智能体级影响加权反事实内在奖励来塑造策略更新。
正文 · 原文
Abstract:Cooperative multi-agent reinforcement learning (MARL) often fails under exogenous variability such as shifts in opponents or environment regimes that cannot be controlled by the team but alter the transition dynamics. We formalize this challenge as Exogenous Dec-POMDP (ED-POMDP), in which an episode-level exogenous regime, selected independently of the team, indexes the dynamics, and the global state is partitioned into components that are directly responsive to the current team action and components that are not directly action-responsive at the current step. This formulation exposes mediated influence: while team actions do not directly update the latter at the current step, they can shape their later evolution through induced changes in the former. Based on this, we propose LEICA, a CTDE-compatible algorithm that learns history-conditioned context representations for both components and shapes policy updates using agent-wise, influence-weighted counterfactual intrinsic rewards. Across SMAX benchmarks with opponent strategy shifts, LEICA consistently improves both training performance and generalization to unseen opponents' strategies over existing baselines, supporting the usefulness of influence-weighted shaping under exogenous train-test regime shifts.
| Comments: | 28 pages. Includes supplementary material. Accepted at NeurIPS 2026 |
| Subjects: | Multiagent Systems (cs.MA); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.22067 [cs.MA] |
| (or arXiv:2611.22067v1 [cs.MA] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22067 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Seongmin Kim [view email]
[v1]
Tue, 17 Nov 2026 01:43:42 UTC (659 KB)
来源:arXiv cs.MA · arxiv.org