arXiv cs.CL· Aman Behera, Namit Solanki, Mehul Anand·· 13 小时前AI 评分29
长上下文问答中的证据位置偏差：Transformer、状态空间与混合 Mamba 架构对比
Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering
AI 导读
研究对比 Transformer、状态空间与混合 Mamba 架构在长上下文问答中的证据位置偏差，将一份金标文档在九个干扰文档间移动。所评估的 Transformer 表现出开头优势，而状态空间模型没有，但两类模型都偏好靠近末尾的证据。该差异仅出现在能完成任务的模型中，且提示词变化可创造或几乎消除开头优势，位置信息在各架构间仍可解码。
正文 · 原文
Abstract:Long-context models often use answer-bearing evidence less effectively when it appears away from a prompt boundary. We compare this position bias across Transformer, state-space, and hybrid Mamba architectures by moving one gold document through a fixed set of nine distractors. The evaluated Transformers show a beginning advantage that the evaluated state-space models lack, while both families favor evidence near the end. A matched pure-versus-hybrid Mamba comparison points in the same direction but remains statistically uncertain. The family difference appears among models that can answer the task. Transformer primacy reappears on synthetic retrieval, where state-space ceiling prevents a family comparison. A corpus swap within one state-space architecture changes overall accuracy without a detectable change in curve shape. Prompt changes can create or nearly remove a beginning advantage. Position remains decodable across architectures even when response curves differ, and attention concentration near initial tokens covaries with beginning sensitivity in the tested Transformer family. These exploratory results show that effective context use depends on the evaluated architecture family, capability, task, and prompt, without isolating the sequence mixer as the sole cause.
| Comments: | Accepted at the 2nd Workshop on Principles of Generative Modeling (PriGM), NeurIPS 2026. 9 pages. Code: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.31135 [cs.CL] |
| (or arXiv:2610.31135v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31135 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Aman Behera [view email]
[v1]
Thu, 22 Oct 2026 11:54:30 UTC (350 KB)
来源：arXiv cs.CL · arxiv.org