arXiv cs.AI· Hui Zhu, Saleh Zare Zade, Rafi Ibn Sultan, Deng Pan, Arun Ross, Dongxiao Zhu·· 10 小时前AI 评分34
BOS Summary:让首个 token 承载输入上下文的无训练推理方法
BOS Summary: The First Token Matters
AI 导读
BOS Summary 是一种无需训练的推理时方法,在 prefill 阶段仅对 BOS query 行放宽因果掩码,使 BOS 聚合全部视觉与文本输入 token 形成上下文表示,且不增加参数、辅助模块或解码开销。
正文 · 原文
Abstract:Large vision-language models (LVLMs) and language models (LLMs) often exhibit attention sinks, whereby particular positions, such as the beginning-of-sequence (BOS) token, receive a disproportionate share of attention. Under standard causal masking, however, the BOS query attends only to itself, so the representation that later tokens repeatedly access carries no information about the input. We introduce BOS Summary, a training-free inference-time method that repurposes this persistent pathway to carry input context. At a single transformer layer during prefill, BOS Summary relaxes the causal constraint for the BOS query row alone, letting BOS aggregate all visual and textual input tokens into a self-derived context representation while every other token retains standard causal attention. The enriched representation then remains accessible to the model throughout decoding, with no additional parameters, auxiliary modules, or decoding-time overhead. Across multiple VLM and LLM families and eight multimodal and text-only benchmarks, BOS Summary improves over direct and Chain-of-Thought prompting and outperforms recent sink-intervention methods on multi-step reasoning tasks. A mechanistic analysis shows that making BOS input-specific also weakens its sink dominance and redistributes attention toward content-bearing prompt tokens, indicating that changing what a sink carries reshapes how later layers allocate attention.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2611.23508 [cs.AI] |
| (or arXiv:2611.23508v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23508 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Dongxiao Zhu [view email]
[v1]
Tue, 17 Nov 2026 21:01:03 UTC (1,470 KB)
来源:arXiv cs.AI · arxiv.org