arXiv cs.CL· Wenjie Zhang, Zhichao Shi, Xuhui Jiang, Xiaojun Wu, Cehao Yang, Chengjin Xu, Xiaoning Song·· 13 小时前AI 评分31
ColaBench：评估个人智能体记忆引导知识获取的新基准
ColaBench: Evaluating Memory-Guided Knowledge Acquisition in Personal Agents
AI 导读
ColaBench 是面向个人智能体的合成基准，覆盖健康与日常生活场景，含 10 个 Profile、349 个任务和 13,033 篇文档的知识库，通过 Evidence Graph 关联个人条件与公开交互中的信息披露。
正文 · 原文
Abstract:Personal agents need to draw on long-term interaction history to retrieve and apply external knowledge suited to a user's current request. Even an answer grounded in relevant documents can be unsuitable when it overlooks the user's circumstances. This creates two connected challenges: using historical conditions to identify the knowledge needed for a new request, and applying that knowledge to the user's situation. Existing benchmarks advance memory recall, personalized retrieval, and memory-guided action, while the joint evaluation of these two challenges remains underexplored. We introduce ColaBench, a synthetic benchmark in health and daily living, with 349 tasks across 10 Profiles and a knowledge base of 13,033 documents. Its Evidence Graph connects personal conditions to their disclosures across public interaction sessions, records prior document exposure, and links answer requirements to personal and newly acquired document evidence. We also introduce MemKB, which uses state summaries to inform queries and rule application, and memory-document links to retrieve knowledge associated with the user's history. MemKB achieves 86.25% Accuracy with an average of 4.07 interaction rounds per task. Task-level analysis finds larger gains on tasks integrating information across multiple historical Facts; error analysis further identifies unsupported personal assumptions and rule misapplication as obstacles to using retrieved evidence. ColaBench provides traceable tasks and an evaluation framework for studying how memory guides knowledge acquisition and rule application.
| Comments: | 12 pages, 4 figures |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.30989 [cs.AI] |
| (or arXiv:2610.30989v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.30989 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Wenjie Zhang [view email]
[v1]
Thu, 22 Oct 2026 09:34:52 UTC (206 KB)
来源：arXiv cs.CL · arxiv.org