arXiv cs.CL· Zichuan Xu, Haochen Zhang, Yuyang Dong, Makoto Onizuka, Chuan Xiao·· 11 小时前AI 评分31
MemShift：将 LLM 派生个人记忆的检测与修复分开评测
MemShift: Separating Detection from Repair in LLM-Derived Personal Memory
AI 导读
MemShift 是一个包含五类变化的基准，用于区分 LLM 派生个人记忆中的检测与修复两个步骤：来源修订、删除、重新定义、写作者切换和导出。该基准在前三类变化中把合成世界分叉为仅变化点不同的分支，金标准值与依赖关系在生成任何文本前固定，并用确定性评分器替代 LLM 评委。
正文 · 原文
Abstract:AI assistants store values that a language model derived from a person's records under the person's own definitions, such as their important contacts. Keeping such a value correct takes two steps: detection, noticing that an input changed, and repair, producing the correct new value. Benchmarks that score only the final answer cannot tell which step failed, and they state every change as new text. Seeing this limitation, we present MemShift, a benchmark of five shifts: source revision, deletion, redefinition, writer switch, and export. The first three fork a synthetic world into branches that differ only in the shift. Gold values and dependencies are fixed before any text, and a deterministic scorer replaces an LLM judge. For a memory that reports its invalidations, MemShift shows for each changed item whether the memory noticed the change and whether its repair was right. Moreover, we build a reference memory with known detection and repair to validate the measure. By evaluating four open-weight LLMs, we find that detection helps only as much as repair allows: the LLMs re-derive few values computed by the person's definitions correctly, and programs that the LLM writes over typed records close most of the gap to exact recomputation. Code and data are available at this https URL.
| Comments: | 54 pages, 9 figures, 32 tables |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.02338 [cs.CL] |
| (or arXiv:2611.02338v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.02338 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zichuan Xu [view email]
[v1]
Mon, 2 Nov 2026 00:35:19 UTC (888 KB)
来源：arXiv cs.CL · arxiv.org