KV-streams：面向智能体强化学习的高效上下文压缩策略
KV-streams for Efficient Compaction in Agentic Reinforcement Learning
研究者提出 KV-streams，一种与任意压缩策略兼容的即插即用方法，通过将 KV cache 向前流式传递而非每次压缩后清空，在三种压缩策略下实现 2.6 至 5 倍的训练墙钟加速。流式 KV cache 还可充当循环状态，携带已从上下文中消失的信息，且仅靠 RL 即可让该行为涌现。
Authors:Emiliano Penaloza, Dane Malenfant, Dheeraj Vattikonda, Roger Creus Castanyer, Siddarth Venkatraman, Abhay Puri, Jonathan Light, Matthew James Sargent, Augustine N. Mavor-Parker, Massimo Caccia, Lucas Caccia, Glen Berseth, Esmeralda S. Whitammer, Alessandro Sordoni, Minseon Kim, Marc-Alexandre Côté, Laurent Charlin, Guillaume Lajoie
Abstract:Scaling the horizon of agentic LLMs is bottlenecked by the need to fit ever longer context traces in GPU memory. Context compaction has been the most popular mechanism to alleviate this issue, keeping GPU memory constant for a given trace. Unfortunately, most compaction strategies rely on prefilling the LLM context many times over, hindering training throughput. To alleviate this bottleneck and enable efficient trainable compaction, we propose KV-streams, a plug-and-play strategy compatible with any compaction strategy that substantially increases throughput while showing no evidence of hindering performance. KV-streams enable scalable compaction by streaming the KV cache forward rather than flushing it after each compaction. We show that KV-streams enable three different compaction strategies, achieving a 2.6 to 5x wall-clock speedup in training. Beyond efficiency, we find that the streamed KV cache can act as a recurrent state, carrying forward information that has long since disappeared from the context. Specifically, in a controlled setting we show that, contrary to prior work, RL alone is all that is needed for this behavior to emerge. Overall, we show KV-streams to be an efficient and lightweight plug-and-play addition to any post-training pipeline.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.35750 [cs.LG] |
| (or arXiv:2609.35750v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.35750 arXiv-issued DOI via DataCite |
Submission history
From: Emiliano Penaloza [view email]
[v1]
Mon, 28 Sep 2026 17:57:42 UTC (17,726 KB)
[v2]
Tue, 29 Sep 2026 17:52:03 UTC (17,691 KB)
[v3]
Thu, 15 Oct 2026 03:45:57 UTC (17,694 KB)
来源：arXiv cs.AI · arxiv.org