arXiv cs.AI· William Hoy, Jingxuan Fan, Nurcin Celik, Xu Pan·· 2 天前AI 评分27
异步对进化策略在长程 LLM 智能体任务上几乎无代价
Asynchrony Is Nearly Free for Evolution Strategies on Long-Horizon LLM Agentic Tasks
AI 导读
研究提出有界陈旧度异步进化策略（bounded-staleness asynchronous ES），在 WebShop 上系统改变策略滞后程度与陈旧适应度测量比例，并设计两项压力测试探测极端陈旧情形。结果显示自然异步几乎与同步 ES 表现持平，即便全部适应度测量均陈旧，评估精度下降仍很小，表明 ES 无需显式离策略校正即可容忍显著策略陈旧，支持其用于长程 LLM 智能体的异步后训练。
正文 · 原文
Abstract:Synchronous policy updates can limit the efficiency of post-training for long-horizon LLM agents. When trajectories span multiple interaction turns, completion times can vary substantially, and synchronous update barriers leave faster workers idling, wasting compute resources. Asynchronous reinforcement learning addresses this inefficiency but introduces policy lag and off-policy optimization. Evolution strategies (ES) are a backpropagation-free alternative to gradient-based training, yet require many perturbed-policy evaluations and have remained largely synchronous in current LLM post-training. We introduce bounded-staleness asynchronous ES and investigate its training behavior under stale fitness measurements. On WebShop, we systematically vary the lag and fraction of stale fitness measurements and additionally introduce two stress tests to probe extreme staleness cases. Natural asynchrony closely tracks synchronous ES, and degradation in evaluation accuracy remains small even under extreme staleness, including when all fitness measurements are stale. These findings suggest that ES can tolerate substantial policy staleness without explicit off-policy correction, supporting its use in asynchronous post-training for long-horizon LLM agents.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.04196 [cs.AI] |
| (or arXiv:2610.04196v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.04196 arXiv-issued DOI via DataCite |
Submission history
From: William Hoy [view email]
[v1]
Sat, 3 Oct 2026 01:18:02 UTC (648 KB)
[v2]
Wed, 14 Oct 2026 16:48:33 UTC (219 KB)
来源：arXiv cs.AI · arxiv.org