用信息价值统一解决强化学习中的廉价检索与观测信用分配

Cheap Retrieval and Observation Credit Assignment from One Quantity: The Value of Information

AI 导读

研究提出用“信息价值”这一单一量同时解决强化学习记忆中的两个问题：检索时无需对整个存储打分，且检索收益的信用可归因到具体观测。该方法借助学习到的世界模型从存档 rollout 中估计信息价值，在观测处读取可给出获取动作最优值的可证下界并抬高其 TD 目标，在决策处读取则定义最优检索集合。一个待用该信号训练的试点检索系统已能以更少上下文完成记忆任务。

正文 · 原文

View PDF HTML (experimental)

Abstract:Memory in reinforcement learning raises two problems. Attention finds the relevant item by scoring the whole store, which is costly for robots under latency and energy limits. When a retrieved memory pays off, credit for acquiring it must cross the whole sequence. We argue that both are governed by one quantity, the value of information, and propose to estimate it from archived rollouts with a learned world model. We obtain that, when read at the observation, it yields a provable lower bound on the optimal value of the acquiring action, which floors its TD target and attributes this floor to the observation that earned it. Read at the decision, it defines the optimal retrieval set. A pilot retrieval system, to be trained and improved with this signal, solves a memory task with reduced context.

Comments:	11 pages, 1 figure. Accepted at the CoRL 2026 Workshop on Memory for Robot Foundation Models (this https URL)
Subjects:	Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
ACM classes:	I.2.6; I.2.8
Cite as:	arXiv:2610.40030 [cs.AI]
	(or arXiv:2610.40030v1 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2610.40030 arXiv-issued DOI via DataCite

Submission history

From: Daniel Strauß [view email]
[v1] Thu, 29 Oct 2026 13:36:06 UTC (26 KB)

来源：arXiv cs.AI · arxiv.org