arXiv cs.CL· Sebastian-Antonio Toma, Jean de Kernier, Mike Ply, Oliviu Matei·· 2 天前AI 评分31
超越下游问答：重新思考如何评估 AI 智能体记忆
Beyond Downstream QA: Rethinking How We Evaluate Agent Memory
AI 导读
研究提出 FEB-30 基准，用 30 条事实演化轨迹（覆盖 6 个诊断类别、3 个难度等级）直接检验智能体记忆存储，而非以下游问答准确率作为代理。
正文 · 原文
Abstract:Agent memory benchmarks evaluate whether an agent's responses reflect correct knowledge, measuring downstream question-answering accuracy as a proxy for memory quality. We show this evaluation paradigm is incomplete: it conflates memory correctness with retrieval and generation quality, producing both false positives where correct responses mask broken memory and false negatives where incorrect responses blame correct memory for retrieval failures. We introduce FEB-30, a benchmark of 30 fact-evolution trajectories spanning 6 diagnostic categories and 3 difficulty levels, constructed via paradigm inversion: documents are generated from known fact trajectories, yielding deterministic ground truth without human annotation. Each trajectory defines how a fact should evolve across multi-author artifacts (documents, emails, conversations), enabling direct inspection of the memory store after each ingestion step. Applying FEB-30 to a representative memory system, we demonstrate that intrinsic and extrinsic evaluation diverge by an average of 16 percentage points, with 44% of trajectories exceeding meaningful divergence thresholds. Diagnostic decomposition reveals that the system correctly classifies memory operations (create, update, reinforce, supersede) only 20% of the time, a failure entirely invisible to extrinsic evaluation, and that failure modes differ systematically by category and difficulty. We release the benchmark, document generation pipeline, and scoring framework to support reproducible evaluation of agent memory systems.
| Comments: | 14 pages, 1 figure, code and data: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.22209 [cs.CL] |
| (or arXiv:2610.22209v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22209 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sebastian-Antonio Toma [view email]
[v1]
Thu, 15 Oct 2026 16:58:12 UTC (38 KB)
来源：arXiv cs.CL · arxiv.org