arXiv cs.AI· Yingxin Jia, Xuanyi Li·· 12 小时前AI 评分31
经验记忆何时真正帮助闭环 LLM 驾驶智能体？一项受控评估
When Does Experience Memory Help Closed-Loop LLM Driving Agents? A Controlled Evaluation with Resampled Warmup Pools, Token-Matched Fillers, and Audited Baselines
AI 导读
一项在 HighwayEnv 测试床（1Hz、gpt-4o-mini）上的受控评估发现，已发表的记忆增益主要来自 warmup 池的抽样运气：报告所用池是十个池中最好的一个，检索记忆仅在十池之一胜过无记忆基线（符号检验 p=0.011）。
正文 · 原文
Abstract:When does experience memory help a closed-loop LLM driving agent? Published gains cannot say, because the two factors we find to matter most, which warmup experiences the pool contains and what any injected tokens do to the prompt, are controlled in none of them. We answer with a controlled evaluation on one audited HighwayEnv testbed (1Hz, gpt-4o-mini for the decisive contrasts). We treat the warmup pool as a resampled random factor (ten disjoint pools) and add token-matched irrelevant-filler controls, oracle-certified diagnostics, caution-pricing metrics, and quantified audits of the baseline ports. The probe suite spans flat-text retrieval (a DiLu port), a single-frame grid port (RESPOND-inspired), our episodic temporal-graph memory, a retrieval-free consolidated rulebook, and a flat K-step observation-history baseline. Four findings follow, each a property of this testbed. (1) Headline memory gains belong to the warmup draw. The reported pool is the best of ten for both retrieval memories, which beat No-Memory in only one pool of ten (sign test p=0.011), and even the rulebook's apparent pool-immunity collapsed on five fresh draws. (2) No retrieval design tested exceeds token-matched filler. The collision-only merge ceiling is reached by a single 15-token caution instruction, and metrics that price caution erase the rulebook's lead. (3) Representational sophistication is nowhere robustly operative. (4) Baseline port quality silently inverts conclusions. On this testbed, a single-pool comparison without filler controls would have reported memory gains that these controls reverse. We release the testbed, audited ports, seeds, and caches.
| Comments: | ACML 2026 |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Robotics (cs.RO) |
| Cite as: | arXiv:2610.27714 [cs.AI] |
| (or arXiv:2610.27714v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.27714 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yingxin Jia [view email]
[v1]
Tue, 20 Oct 2026 06:31:27 UTC (189 KB)
来源：arXiv cs.AI · arxiv.org