arXiv cs.CL· Zeyu Liu, Jian Zhong, Rongduo Han, Ziyang Wu, Shunye Tang, Chenghao He, Yaxuan Yang, Yihang Qiu, Ailing Wang, Xiao Liang, Guohuan Xie, Xiaokang Xue, Gongchen Li, Haining Zhang, Wei Wang·· 12 小时前AI 评分32
EvalMem：面向长期记忆系统的操作级诊断框架
EvalMem: An Operation-Level Diagnostic Framework for Long-Term Memory Systems
AI 导读
EvalMem 是一个操作级诊断框架，通过 Encoding、Retrieval、Generation 三个并行 Examiner 分别检查目标事实是否被存储、原生检索器能否返回可用证据、模型能否基于 oracle 证据作答，输出细粒度多标签缺陷码。
正文 · 原文
Authors:Zeyu Liu, Jian Zhong, Rongduo Han, Ziyang Wu, Shunye Tang, Chenghao He, Yaxuan Yang, Yihang Qiu, Ailing Wang, Xiao Liang, Guohuan Xie, Xiaokang Xue, Gongchen Li, Haining Zhang, Wei Wang
Abstract:Long-horizon interactions with LLM-based assistants require memory systems that preserve and update user states, preferences, and interaction histories. Existing evaluations report end-to-end QA accuracy and cannot determine whether errors arise from encoding, retrieval, or generation. We introduce EvalMem, an operation-level diagnostic framework with three parallel Examiners. For each query, the Encoding Examiner checks whether the target fact is stored, the Retrieval Examiner assesses whether the native retriever returns usable evidence, and the Generation Examiner tests whether the model can answer from oracle evidence. Their outputs form fine-grained multi-label defect codes. To improve store-level diagnosis, we adapt agentic RAG with a recall-first strategy that searches using both the query and source evidence, increasing recall of present evidence on LoCoMo from 70.2% to 95.6%. Evaluations of seven memory systems on LoCoMo, LongMemEval-S, and dynamic DynaMem-Bench identify retrieval as the most frequently attributed failure layer; in default LoCoMo, retrieval defects reach 22.1%, compared with 7.7% for encoding and 6.5% for generation. Guided by this diagnosis, MemWiki, a search-friendly auxiliary structure built from each system's memory export, improves mean accuracy by 2.5 and 2.3 percentage points on LoCoMo and LongMemEval-S, respectively.
| Comments: | Accepted by EMNLP 2026 Findings |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2609.22231 [cs.CL] |
| (or arXiv:2609.22231v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.22231 arXiv-issued DOI via DataCite |
Submission history
From: Jian Zhong [view email]
[v1]
Thu, 3 Sep 2026 04:20:45 UTC (1,574 KB)
[v2]
Mon, 2 Nov 2026 09:37:40 UTC (1,575 KB)
来源：arXiv cs.CL · arxiv.org