arXiv cs.CL· Zihe Song, Wenting Zhao, Ye Liu, Yuanjie Zhu, Ke Xu, Shanghao Li, Philip S. Yu·· 12 小时前AI 评分38
SpecMem：通过实时记忆管理加速智能体工具调用
SPECMEM: Accelerating Agentic Tool Calling via Live Memory Management
AI 导读
SpecMem 提出一种在部署期间持续维护的工具调用记忆机制，将已验证的调用在线写回带 LRU 淘汰的有界用户级存储，并用副作用门控延迟不可逆调用。在三个开源权重模型的多会话多用户查询流上，SpecMem 相比冻结存储将平均接受 token 数提升 27%（各基准 +11% 至 +34%），实际耗时最高加速 1.74 倍。
正文 · 原文
Abstract:Tool-calling agents are mostly optimized for accuracy; in deployment, however, their growing cost is latency: every tool call is decoded autoregressively. Speculative decoding accelerates this without changing the output, and retrieved past calls make strong drafts under real use, where each user's behavior tends to repeat. Yet existing systems keep no cross-session memory, or build the draft store once and never update it. We propose SpecMem, a tool-call memory kept alive and managed throughout deployment: verified calls are written back online into bounded per-user stores with LRU eviction, and a side-effect gate defers irreversible calls until verification confirms. On multi-session, multi-user query streams served by three open-weight models, a frozen store steadily loses acceptance while SpecMem holds it: a 27% gap in mean accepted tokens (+11% to +34% across benchmarks, tracking how much novel work arrives after the store is built) and up to 1.74x wall-clock speedup. Ablations attribute the gain almost entirely to keeping the store live; local per-user scoping adds no acceptance but closes a measured cross-user leakage channel, and gating on declared side effects, rather than on retrieval confidence, is what eliminates irreversible mistakes.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.31840 [cs.CL] |
| (or arXiv:2610.31840v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31840 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zihe Song [view email]
[v1]
Thu, 22 Oct 2026 19:24:21 UTC (117 KB)
来源：arXiv cs.CL · arxiv.org