arXiv cs.CL· Jipei He, Wenhui Tan, Xiaoyi Yu, Enver Sangineto, Fiorenzo Parascandolo, Rita Cucchiara, Ruihua Song·· 10 小时前AI 评分39
STAIR:利用前序问题计算提升 LLM 后续解题能力
Can Computation from Earlier Problems Help LLMs Solve New Ones?
AI 导读
研究者提出 STAIR(Stale-Token Attention for Inter-query Reuse),将前序回答生成中的 attention key 与 value 存入只读 bank,仅训练 12,288 个参数,通过学习反射当前 query 向量来重定向对该 bank 的注意力。
正文 · 原文
Abstract:Large language models often solve independent problems in the same conversation. Can computation from earlier problems help them solve new ones? Preliminary experiments show that retained history can raise or lower later-turn accuracy, even within the same domain. To trace this effect, we hold each problem fixed and vary its preceding history. The resulting changes in the model's internal states contain a recurring, problem-dependent component. To improve reasoning with retained history, we introduce STAIR (Stale-Token Attention for Inter-query Reuse). It captures attention keys and values from earlier response generation in a read-only bank. STAIR learns to reflect current query vectors to redirect attention over this bank during prompt processing. The backbone and stored entries remain fixed; only 12,288 parameters are trained. Across three Qwen models and four reasoning benchmarks, STAIR improves average later-turn accuracy by up to 11.67 percentage points over the unmodified model with history. The math-trained Qwen3.5-9B controller also improves code generation on HumanEval+ without further training.
| Comments: | 42 pages, 8 figures. v2: Expanded experiments and analyses; revised presentation |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2609.39394 [cs.AI] |
| (or arXiv:2609.39394v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.39394 arXiv-issued DOI via DataCite |
Submission history
From: Jipei He [view email]
[v1]
Wed, 30 Sep 2026 09:43:16 UTC (470 KB)
[v2]
Thu, 19 Nov 2026 11:31:27 UTC (562 KB)
来源:arXiv cs.CL · arxiv.org