arXiv cs.AI· Jiaqi Shao, Yiyi Lu, Yunzhen Zhang, Bing Luo·· 14 小时前AI 评分40
智能体记忆的规模条件化评估：无关会话累积下证据何时失效
When Stored Evidence Stops Being Usable: Scale-Conditioned Evaluation of Agent Memory
AI 导读
研究提出一种规模条件化评估协议，在保持任务证据固定的前提下持续加入无关会话，用四项诊断指标衡量智能体记忆的可用性边界。在 LongMemEval 上，HippoRAG 虽未超出两次调用预算，但预算合规可靠性下降 16-20 个百分点；LiCoMemory 的失败则强依赖智能体，Qwen3-8B 超出预算而 Qwen3-32B 与 Qwen3-235B 在测试范围内保持可靠。
正文 · 原文
Abstract:Memory-agent evaluations report fixed-snapshot accuracy or retrieval quality, but these scores do not show whether evidence remains usable as irrelevant sessions (sessions not annotated as task-relevant evidence for the query) accumulate. We present a scale-conditioned evaluation protocol for agent memory under evidence-preserving growth: for each query, task evidence is held fixed while irrelevant sessions are added. The protocol logs agent-memory trajectories and reports four diagnostics: budget-compliant reliability, tail memory-call burden, failure-regime decomposition, and the usable-scale boundary where reliability falls below the target. Applied to LongMemEval and LoCoMo across flat, planar, and hierarchical memory interfaces, the protocol shows reliability loss is not a single phenomenon. On LongMemEval, HippoRAG stays within the two-call budget but loses 16-20 percentage points in budget-compliant reliability as irrelevant sessions are added; LiCoMemory's observed failures depend strongly on the agent, with Qwen3-8B exceeding the budget while Qwen3-32B and Qwen3-235B remain reliable in the tested range. The result supports a framework for making scalable-memory claims conditional on agent, interface, scale range, and interaction budget. Code and evaluation artifacts: this https URL
| Comments: | Accepted at NeurIPS 2026, Evaluations and Datasets Track (poster). Updated to the camera-ready version. 25 pages, including references and appendices |
| Subjects: | Artificial Intelligence (cs.AI) |
| ACM classes: | I.2.7; H.3.3 |
| Cite as: | arXiv:2605.07313 [cs.AI] |
| (or arXiv:2605.07313v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.07313 arXiv-issued DOI via DataCite |
Submission history
From: Yiyi Lu [view email]
[v1]
Fri, 8 May 2026 06:22:31 UTC (1,169 KB)
[v2]
Thu, 29 Oct 2026 07:34:59 UTC (1,355 KB)
来源：arXiv cs.AI · arxiv.org