arXiv cs.AI· Xiang Li, Nan Jiang·· 12 小时前AI 评分31
Q-MMR：基于递归重加权与矩匹配的离策略评估框架
Q-MMR: Off-Policy Evaluation via Recursive Reweighting and Moment Matching
AI 导读
Q-MMR 提出一种有限时域 MDP 下的离策略评估理论框架，为每个数据点学习标量权重，使重加权奖励逼近目标策略的期望回报，权重通过针对价值函数判别器类的矩匹配目标以自顶向下方式归纳学习。该框架仅需 Q^π 可实现性即可建立与维度无关的有限样本保证，误差不依赖函数类的统计复杂度。研究还建立了其与重要性采样、线性 FQE 等现有方法的联系，并重新审视离线 RL 中覆盖性的本质。
正文 · 原文
Abstract:We present a novel theoretical framework, Q-MMR, for off-policy evaluation in finite-horizon MDPs. Q-MMR learns a set of scalar weights, one for each data point, such that the reweighted rewards approximate the expected return under the target policy. The weights are learned inductively in a top-down manner via a moment matching objective against a value-function discriminator class. Notably, and perhaps surprisingly, a data-dependent finite-sample guarantee for general function approximation can be established under only the realizability of $Q^\pi$, with a dimension-free bound -- that is, the error does not depend on the statistical complexity of the function class. We also establish connections to several existing methods, such as importance sampling and linear FQE. Further theoretical analyses shed new light on the nature of coverage, a concept of fundamental importance to offline RL.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2605.06474 [cs.LG] |
| (or arXiv:2605.06474v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.06474 arXiv-issued DOI via DataCite |
Submission history
From: Xiang Li [view email]
[v1]
Thu, 7 May 2026 16:00:04 UTC (122 KB)
[v2]
Fri, 8 May 2026 12:50:37 UTC (122 KB)
[v3]
Fri, 23 Oct 2026 17:57:34 UTC (128 KB)
来源：arXiv cs.AI · arxiv.org