arXiv cs.CL· Jiangze Yan, Yi Shen, Wenjing Zhang, Jieyun Huang, Ping Chen, Zhaoxiang Liu, Ning Wang, Kai Wang, Shiguo Lian·· 3 天前AI 评分23
HiMPO：面向长时程智能体的后见之明记忆策略优化，降低信用纠缠
HiMPO: Hindsight-Informed Memory Policy Optimization for Less-Entangled Credit in Long-Horizon Agents
AI 导读
研究者提出 HiMPO，一种后见之明记忆策略优化框架，在轨迹级强化学习之外为记忆写入单独分配信用：每次更新时固定其余写入前上下文，比较目标答案在旧、新记忆下的对数似然，并用基于组相对后见相关性的软门控加权局部效用优势。在 Local Wiki Search 上，其平均 F1 与精确匹配较 MemPO 分别提升 2.6 和 2.2 个百分点，同时保持压缩上下文效率。
正文 · 原文
Abstract:Long-horizon agents compress interaction histories into memory, but final outcomes entangle memory-writing credit with tool failures and reasoning errors. We propose HiMPO, a Hindsight-Informed Memory Policy Optimization framework that supplements trajectory-level reinforcement learning with memory-specific credit. For each update, HiMPO compares the target answer's log-likelihood under the previous and updated memories while holding the remaining pre-write context fixed. A soft gate based on group-relative hindsight relevance then weights the local utility advantage. Trajectory-level credit applies to all generated tokens; memory tokens additionally receive the auxiliary advantage. Across judge-based open-domain tasks and objective compressive-memory QA, HiMPO improves task performance while preserving compressed-context efficiency. On Local Wiki Search, it improves average F1 and exact match over MemPO by 2.6 and 2.2 percentage points, respectively. Controlled interventions and live memory-reversion studies provide evidence of reduced blame leakage and alignment between memory credit and behavioral effects. With the policy fixed, the credit estimator remains robust to the tested forms of target noise.
| Comments: | Preprint. 2 figures |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2606.16285 [cs.CL] |
| (or arXiv:2606.16285v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.16285 arXiv-issued DOI via DataCite |
Submission history
From: Jiangze Yan [view email]
[v1]
Mon, 15 Jun 2026 06:49:18 UTC (2,897 KB)
[v2]
Tue, 1 Sep 2026 03:33:12 UTC (489 KB)
[v3]
Wed, 14 Oct 2026 01:37:07 UTC (493 KB)
来源：arXiv cs.CL · arxiv.org