arXiv cs.CL· Shiwen Qin, Cameron Barker, Henry Gouk, Elliot J. Crowley, Shay B. Cohen, Mirella Lapata, Miao Li·· 2 天前AI 评分31
超越上下文长度：刻画长上下文推理的难度
Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning
AI 导读
研究者构建了 LURCH 基准，包含 6,550 个多领域问答实例及支撑证据与推理图，用证据范围、证据分散度和推理深度三个属性量化长上下文推理难度。对闭源与开源权重模型的评测显示，在固定上下文长度下，这三个属性均与模型表现强负相关；上下文变长时，证据分散的样本性能急剧下降，而证据集中的样本相对稳定。三个属性还会叠加放大难度，三者均中等偏高的题目比仅一项极端的题目更难。
正文 · 原文
Abstract:Long-context language models support ever longer context windows, and evaluations typically treat context length as the main indicator of difficulty. However, context length alone cannot explain why models fail on long-context tasks that require reasoning over complex information. Contexts of equal length can pose very different challenges: a task may be solvable from a single local passage, or it may require integrating evidence distributed across the full context. We quantitatively characterise this difficulty through three properties: evidence scope, how much necessary information must be found; evidence dispersion, how widely that information is scattered within the context; and reasoning depth, how many dependent steps are needed to derive an answer from it. We construct LURCH, a multi-domain question-answering benchmark of 6,550 instances with supporting evidence and reasoning graphs, enabling all three properties to be quantified. Evaluations of proprietary and open-weight models show that: (1) at fixed context length, all our three properties are strongly negatively correlated with model performance; (2) as context length increases, performance degrades sharply on samples with dispersed evidence, whereas it remains relatively stable on samples with localised evidence; and (3) difficulty compounds across properties, as questions with moderately high values on all three are harder than those with an extreme value on just one. These findings establish evidence scope, dispersion and reasoning depth as measures of long-context reasoning difficulty that complement context length.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.22067 [cs.CL] |
| (or arXiv:2610.22067v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22067 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shiwen Qin [view email]
[v1]
Thu, 15 Oct 2026 15:28:33 UTC (423 KB)
来源：arXiv cs.CL · arxiv.org