arXiv cs.AI· Yanjie Dong, Zhou Su, Wei Wang, Gang Wang·· 12 小时前AI 评分18
基于马尔可夫轨迹的重要性修正动量 Actor-Critic 统一分析
A Unified Analysis of Importance-Corrected Momentum Actor-Critic over a Markovian Trajectory
AI 导读
研究提出一种通用 actor-critic 方法，将岭正则化路径式 LSTD 评论家与重要性修正动量（ICM）演员结合，评论家通过 Newton-Schulz 求逆逐帧求解，无需步长、混合时间知识和秩条件。
正文 · 原文
Abstract:We present a generic actor-critic method that combines a ridge-regularized pathwise least-squares temporal-difference (LSTD) critic with importance-corrected momentum (ICM) actors. The critic solves a ridge-regularized system per frame by Newton-Schulz inversion and needs no stepsize, no knowledge of the mixing time, and no rank condition. Averaging stochastic policy gradients over frames of $T$ consecutive transitions, the ICM actors incur a frame-dependent gradient bias of order $\frac{1}{T}$. To our knowledge, we establish the first finite-time bounds for ICM actor-critic methods with normalized and unnormalized steps along a single Markovian trajectory. Up to a stationary-to-discounted distribution mismatch, the averaged gradient norm over $K$ updates is bounded by ${\cal O}(K^{-1/3}+\frac{1}{T})$ for normalized steps and by ${\cal O}(K^{-1/4}+\frac{1}{T})$ for unnormalized steps. We specialize these bounds to Muon, normalized stochastic gradient descent (SGD) with momentum, SGD with momentum, and AdaNorm, and compare them in the nuclear norm. With a dimension-free stepsize, Muon confines the factor $\min\{m,n\}^{1/2}$ associated with an $m\times n$ actor parameter matrix to the noise and correction terms. The comparison also characterizes when orthogonalization guarantees a larger noiseless ascent than normalization: exactly when the gradient's nuclear rank exceeds the spectral-to-Frobenius curvature ratio.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.33477 [cs.LG] |
| (or arXiv:2610.33477v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33477 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yanjie Dong [view email]
[v1]
Sat, 24 Oct 2026 06:03:53 UTC (55 KB)
来源：arXiv cs.AI · arxiv.org