arXiv cs.AI· Ku Onoda, Paavo Parmas, Manato Yaguchi, Yutaka Matsuo·· 15 小时前AI 评分26
可微模拟器真能给出更好的策略梯度吗？ICLR 2026 论文重新审视该问题
Does "Do Differentiable Simulators Give Better Policy Gradients?" Give Better Policy Gradients?
AI 导读
针对可微模拟器中不连续动力学导致一阶策略梯度估计有偏的问题，论文提出 DDCG——仅需单个超参数的轻量检测方法，在不平滑区域切换估计器，小样本下仍保持稳健。在可微机器人控制任务上，作者另提出逐步逆方差实现 IVW-H，无需显式不连续检测即可稳定方差并取得强结果。结论是估计器切换提升了受控实验的稳健性，但实际部署中方差控制往往更关键。
正文 · 原文
Abstract:In policy gradient reinforcement learning, access to a differentiable model enables 1st-order gradient estimation that accelerates learning compared to relying solely on derivative-free 0th-order estimators. However, discontinuous dynamics cause bias and undermine the effectiveness of 1st-order estimators. Prior work addressed this bias by constructing a confidence interval around the REINFORCE 0th-order gradient estimator and using these bounds to detect discontinuities. However, the REINFORCE estimator is notoriously noisy, and we find that this method requires task-specific hyperparameter tuning and has low sample efficiency. This paper asks whether such bias is the primary obstacle and what minimal fixes suffice. First, we re-examine standard discontinuous settings from prior work and introduce DDCG, a lightweight test that switches estimators in nonsmooth regions; with a single hyperparameter, DDCG achieves robust performance and remains reliable with small samples. Second, on differentiable robotics control tasks, we present IVW-H, a per-step inverse-variance implementation that stabilizes variance without explicit discontinuity detection and yields strong results. Together, these findings indicate that while estimator switching improves robustness in controlled studies, careful variance control often dominates in practical deployments.
| Comments: | ICLR2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO) |
| Cite as: | arXiv:2604.18161 [cs.LG] |
| (or arXiv:2604.18161v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.18161 arXiv-issued DOI via DataCite |
|
| Journal reference: | The Fourteenth International Conference on Learning Representations. ICLR 2026 |
Submission history
From: Ku Onoda [view email]
[v1]
Mon, 20 Apr 2026 12:23:48 UTC (6,225 KB)
[v2]
Sat, 24 Oct 2026 00:48:22 UTC (6,213 KB)
来源：arXiv cs.AI · arxiv.org