跳到正文
arXiv cs.CL· Mingchen Li, Jiatan Huang, Chuxu Zhang, Liang Zhao, Hong Yu·· 9 小时前AI 评分34

将 RAG 视为上下文优化:从梯度下降视角实现冻结 LLM 的轻量适配

In-Context Optimization for Retrieval-Augmented Generation: A Gradient-Descent Perspective

AI 导读

研究将检索增强生成(RAG)视为上下文优化过程,证明单层线性自注意力可对统一线性化 RAG 目标执行一步梯度下降,使检索增强预测与上下文优化在精确条件下重合。基于此提出一种面向冻结 RAG LLM 的轻量方法:保持检索器与主干固定,仅预测生成器侧证据使用接口的上下文条件更新。

正文 · 原文

View PDF HTML (experimental)

Abstract:In-context learning has recently been linked to implicit gradient descent in linear self-attention models, suggesting that context can induce a forward-pass update. Retrieval-augmented generation (RAG) also relies on context, but retrieved documents are usually treated as static evidence rather than signals for adaptation. We study RAG as an in-context optimization process. First, we show that one linear self-attention layer can implement one gradient-descent step on a unified linearized RAG objective covering both projection-based and dot-product retrieval interfaces. This gives an exact regime where retrieval-augmented prediction and in-context optimization coincide. We use this result not as a literal model of LLM computation, but as a guide for adapting the interaction between queries and retrieved evidence. We then test the boundary of this correspondence: it remains stable under controlled linear extensions, but becomes feature-distribution dependent under nonlinear architectures. Finally, we turn this view into a lightweight method for frozen RAG LLMs. The method keeps the retriever and backbone fixed, and predicts a context-conditioned update to a generator-side evidence-use interface. Across seven QA benchmarks, two retrievers, and two frozen LLM backbones, this forward-only update improves a shared-interface baseline, transfers to held-out tasks, and approaches test-time gradient adaptation at much lower per-query cost.
Comments: NIPS 2026
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2605.26356 [cs.CL]
  (or arXiv:2605.26356v2 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2605.26356

arXiv-issued DOI via DataCite

Submission history

From: Mingchen Li [view email]
[v1] Mon, 25 May 2026 22:04:54 UTC (273 KB)
[v2] Tue, 17 Nov 2026 18:19:55 UTC (302 KB)

来源:arXiv cs.CL · arxiv.org