arXiv cs.AI· Tao Ren, Hui Yang, Jia Deng, Tingyang Chen, Qiyao Wang, Shuo Tang, Ziyang Zeng, Yuhua Jiang, Guanghao Li, Huatong Song, Yining Li, Yifan Luo, Kunbao Ding, Feng Chang, Chuan Hao, Ran Tao, Bryan Dai, Jiafeng Liang, Jinyang Jiang·· 15 小时前AI 评分34
MPO：面向长程 LLM 智能体的多尺度策略优化
Multiscale Policy Optimization for Long-Horizon LLM Agents
AI 导读
研究者提出多尺度策略优化（MPO），结合多尺度优势估计（MAE）与轮次级重要性采样，将信用分配与策略更新对齐到轮次尺度。在搜索、工具使用和自动化任务上，4B 与 30B-A3B 模型的实验均稳定优于强 RL 基线。
正文 · 原文
Authors:Tao Ren, Hui Yang, Jia Deng, Tingyang Chen, Qiyao Wang, Shuo Tang, Ziyang Zeng, Yuhua Jiang, Guanghao Li, Huatong Song, Yining Li, Yifan Luo, Kunbao Ding, Feng Chang, Chuan Hao, Ran Tao, Bryan Dai, Jiafeng Liang, Jinyang Jiang
Abstract:Long-horizon LLM agents interact with their environments through complete assistant turns while generating each action token by token. We observe that critic values vary primarily across turns, motivating optimization anchored at turns with finer credit allocation among tokens. We propose Multiscale Policy Optimization (MPO), combining Multiscale Advantage Estimation (MAE) and turn-level importance sampling to align credit assignment and policy updates at the turn scale. Extensive experiments across search, tool-use, and automation tasks with 4B and 30B-A3B models demonstrate consistent gains over strong RL baselines.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.35013 [cs.AI] |
| (or arXiv:2610.35013v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35013 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tao Ren [view email]
[v1]
Mon, 26 Oct 2026 08:22:11 UTC (1,482 KB)
来源：arXiv cs.AI · arxiv.org