arXiv cs.CL· Mingchen Li, Jiatan Huang, Chuxu Zhang, Liang Zhao, Hong Yu·· 9 小时前AI 评分34
将 RAG 视为上下文优化:从梯度下降视角实现冻结 LLM 的轻量适配
In-Context Optimization for Retrieval-Augmented Generation: A Gradient-Descent Perspective
AI 导读
研究将检索增强生成(RAG)视为上下文优化过程,证明单层线性自注意力可对统一线性化 RAG 目标执行一步梯度下降,使检索增强预测与上下文优化在精确条件下重合。基于此提出一种面向冻结 RAG LLM 的轻量方法:保持检索器与主干固定,仅预测生成器侧证据使用接口的上下文条件更新。
正文 · 原文
Abstract:In-context learning has recently been linked to implicit gradient descent in linear self-attention models, suggesting that context can induce a forward-pass update. Retrieval-augmented generation (RAG) also relies on context, but retrieved documents are usually treated as static evidence rather than signals for adaptation. We study RAG as an in-context optimization process. First, we show that one linear self-attention layer can implement one gradient-descent step on a unified linearized RAG objective covering both projection-based and dot-product retrieval interfaces. This gives an exact regime where retrieval-augmented prediction and in-context optimization coincide. We use this result not as a literal model of LLM computation, but as a guide for adapting the interaction between queries and retrieved evidence. We then test the boundary of this correspondence: it remains stable under controlled linear extensions, but becomes feature-distribution dependent under nonlinear architectures. Finally, we turn this view into a lightweight method for frozen RAG LLMs. The method keeps the retriever and backbone fixed, and predicts a context-conditioned update to a generator-side evidence-use interface. Across seven QA benchmarks, two retrievers, and two frozen LLM backbones, this forward-only update improves a shared-interface baseline, transfers to held-out tasks, and approaches test-time gradient adaptation at much lower per-query cost.
| Comments: | NIPS 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.26356 [cs.CL] |
| (or arXiv:2605.26356v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.26356 arXiv-issued DOI via DataCite |
Submission history
From: Mingchen Li [view email]
[v1]
Mon, 25 May 2026 22:04:54 UTC (273 KB)
[v2]
Tue, 17 Nov 2026 18:19:55 UTC (302 KB)
来源:arXiv cs.CL · arxiv.org