面向 LLM 智能体的扰动鲁棒策略学习：Stable Perturbation-Robust Policy Optimization（SPrPO）

Learning Perturbation Robust Policies for LLM Agents with Stable Optimization

AI 导读

研究提出 Stable Perturbation-Robust Policy Optimization（SPrPO），在强化学习训练中对 LLM 智能体施加自适应、敏感度感知的扰动，以提升策略对隐藏状态噪声、剪枝和量化等扰动的鲁棒性。该方法基于对扰动策略更新保持稳定单调改进条件的分析，在 ALFWorld 和 WebShop 上跨多种扰动类型与规模验证，实现了鲁棒性提升且策略优化保持稳定。

正文 · 原文

View PDF HTML (experimental)

Abstract:Reinforcement learning (RL) has become an effective post-training paradigm for long-horizon large language model (LLM) agents. However, we find that the resulting policies can be sensitive to various policy perturbations, such as hidden-state noise, pruning, and quantization. In this work, we study how to improve perturbation robustness during policy optimization. We first introduce the notion of a perturbation robust policy and analyze conditions under which perturbed policy updates preserve stable monotonic improvement. Based on this analysis, we introduce Stable Perturbation-Robust Policy Optimization (SPrPO), which applies adaptive and sensitivity-aware perturbations during RL training. We evaluate SPrPO on ALFWorld and WebShop and conduct systematic experiments across multiple perturbation types and scales, showing improved perturbation robustness while maintaining stable policy optimization.

Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2609.34064 [cs.LG]
	(or arXiv:2609.34064v2 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2609.34064 arXiv-issued DOI via DataCite

Submission history

From: Pengxin Wang [view email]
[v1] Mon, 28 Sep 2026 00:43:16 UTC (5,674 KB)
[v2] Tue, 10 Nov 2026 05:09:29 UTC (5,674 KB)

来源：arXiv cs.AI · arxiv.org