arXiv cs.MA· Matthias Gerstgrasser, Gianluca Brero, Alon Eden, Darshan Chakrabarti, Nicolas Lepore, Vincent Li, Eric Mibuari, Amy Greenwald, David C. Parkes·· 9 天前AI 评分22
Stackelberg POMDP：通过强化学习学会领导
Stackelberg POMDP: Learning to Lead via Reinforcement Learning
AI 导读
研究者提出 Stackelberg POMDP 框架，将跟随者的策略适应嵌入领导者环境，把部分可观测的多跟随者领导-跟随问题转化为单智能体 POMDP，并证明仅基于领导者博弈历史的最优策略可实现最优承诺。在平台设计中，学到的展示规则使平均消费者剩余比优化的固定价格上限提高 8.4%；在间接机制设计中，使用买方消息的机制在所有测试类型数下社会福祉均高于最优标准序贯价格机制。
正文 · 原文
Abstract:Many real-world domains--including e-commerce platform design, security planning, and multi-agent coordination--feature leader-follower problems where one decision-maker commits to a policy and others react strategically. We develop a reinforcement learning framework for such interactions in sequential environments with partial observations and multiple followers. Followers may adapt through no-regret learning or reinforcement learning, potentially departing from equilibrium behavior. The framework embeds follower adaptation into the leader's environment to construct a single-agent partially observable Markov decision process--the Stackelberg POMDP. For policy-interactive response algorithms, which access the leader's policy through queries, we prove that an optimal policy based only on the leader's game history yields an optimal commitment under the specified response procedure. We use proximal policy optimization with a centralized critic and train contextual meta-followers to respond across leader policies. In indirect mechanism design, mechanisms using buyer messages achieve higher social welfare than optimal standard sequential price mechanisms across all tested type counts, with responses certified as approximate Bayesian coarse correlated equilibria. In platform design, learned display rules increase mean consumer surplus by 8.4% over an optimized fixed price cap while accommodating hidden seller costs. In Atari bilateral trade, meta-learned follower responses support joint learning of visual gameplay and economic decisions; assigning leadership to the seller or buyer shifts transaction prices and payoffs in that agent's favor. Controlled ablations examine how response credit, policy consistency, and reward timing affect learning.
| Subjects: | Computer Science and Game Theory (cs.GT); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2210.03852 [cs.GT] |
| (or arXiv:2210.03852v5 [cs.GT] for this version) | |
| https://doi.org/10.48550/arXiv.2210.03852 arXiv-issued DOI via DataCite |
Submission history
From: Gianluca Brero [view email]
[v1]
Fri, 7 Oct 2022 23:55:51 UTC (2,857 KB)
[v2]
Wed, 15 Feb 2023 17:18:32 UTC (3,326 KB)
[v3]
Fri, 10 Nov 2023 00:36:59 UTC (4,073 KB)
[v4]
Fri, 19 Jul 2024 11:35:43 UTC (375 KB)
[v5]
Thu, 8 Oct 2026 00:10:11 UTC (293 KB)
来源：arXiv cs.MA · arxiv.org