NVIDIA 如何为高效长上下文推理协同设计稀疏注意力
Co-Designing Sparse Attention for Efficient Long-Context Inference
NVIDIA 技术博客探讨了面向长上下文推理的稀疏注意力协同设计:当上下文从数万 token 扩展到数十万 token 时,稠密注意力因每个 query 都要与所有 key 计算而成为 Transformer 层中最昂贵的部分,在 128K token 下可能占据大部分 prefill 时间。文章围绕这一瓶颈讨论软硬件协同的稀疏注意力方案。
Agentic workloads, long documents, and large codebases are pushing context lengths from tens of thousands of tokens to hundreds of thousands. At those lengths,...
Agentic workloads, long documents, and large codebases are pushing context lengths from tens of thousands of tokens to hundreds of thousands. At those lengths, attention becomes the most expensive part of a transformer layer. Dense attention scores every query against every key, so its cost per query grows with the context length N, and at 128K tokens it can take most of the prefill time in a…
来源:NVIDIA Technical Blog · developer.nvidia.com