arXiv cs.AI· Karol Struniawski·· 10 小时前AI 评分29
集中式与弥散式注意力头丢弃下自回归多步推理的退化差异
Breakdown of Autoregressive Multi-Step Reasoning Under Concentrated vs. Diffuse Attention Head Dropout
AI 导读
在 GSM8K 的 250 组配对推导上，集中爆发的注意力头掩码比同等预算的弥散 i.i.d. dropout 造成更严重的推理退化：Qwen3-8B 准确率从 91.2% 降至 78.8%（-12.4 pp，p=5.5×10⁻⁶），LLaMA-3.1-8B-Instruct 从 84.0% 降至 76.8%（-7.2 pp，p=0.0198）。
正文 · 原文
Abstract:During extended wakefulness, mammalian neocortical networks experience localized periods of transient neuronal silence termed off-states. Here, we examine how analogous spatiotemporally concentrated disruptions affect sequential reasoning in autoregressive Large Language Models (LLMs). Evaluating 250 paired derivations on the GSM8K benchmark across two 8B models (Qwen3-8B and LLaMA-3.1-8B-Instruct), we contrast clustered bursts of attention head masking against budget-matched diffuse i.i.d. dropout. Concentrated bursts induce substantial reasoning degradation: at onset parameter $\mathcal{B}=0.10$ (scheduling $\sim 5.7\%$ of vulnerable head-token volume), Qwen3-8B accuracy drops from $91.2\%$ under diffuse noise to $78.8\%$ under burst dropout ($\Delta = -12.4$ percentage points [pp], exact McNemar $p = 5.5 \times 10^{-6}$), while LLaMA-3.1-8B-Instruct drops from $84.0\%$ to $76.8\%$ ($\Delta = -7.2$ pp, $p = 0.0198$). Conversely, dispersing the identical cumulative budget uniformly across generation steps via stratified quintile matching yields substantially milder degradation (retaining $91.2\%$ in Qwen and $84.0\%$ in LLaMA). In an exploratory factorial dissociation ($N=100$), single-step dropout density is associated with an acute drop in a single schedule draw ($-10.0$ pp on Qwen, $p_{\text{holm}} = 0.0425$), while multi-token continuity exhibits an inconclusive compounding trend ($-7.0$ pp, $p_{\text{holm}} = 0.1892$). Error analysis reveals that Qwen failures reflect derivation errors within formatted responses (conditional accuracy drop of $-11.4$ pp), whereas in LLaMA $22.2\%$ of discordant losses lack a final answer format line (conditional gap narrows to $-5.0$ pp). These findings demonstrate that autoregressive reasoning chains are particularly vulnerable to concentrated structural disruptions, providing empirical benchmarks for fault tolerance in transformer architectures.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.14209 [cs.AI] |
| (or arXiv:2611.14209v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14209 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Karol Struniawski PhD [view email]
[v1]
Tue, 10 Nov 2026 15:29:57 UTC (129 KB)
来源：arXiv cs.AI · arxiv.org