arXiv cs.CL· Alexandros Christoforos, Chadbourne Davis, Paul Raj·· 11 小时前AI 评分31
双重预算下的稀疏文本扩散：上下文连通性与专家激活分析
Sparse Text Diffusion under Dual Budgets: Context Connectivity and Expert Activation
AI 导读
研究将长上下文扩散模型的成本拆为注意力图可用边数与每 token 激活专家数两项预算，提出 DualBudget 稀疏专家配置，并在科学摘要、多跳问答、对话和句对预测任务上分析。长度实验显示在 16K 以内优于所列基线，但随长度增加出现可测的性能退化；成本记录显示相对 DiffuSeq 降低 20%，但计时约定未说明。
正文 · 原文
Abstract:Long-context diffusion repeatedly pays for both token interaction and feed-forward computation. We formulate these costs as two separate budgets: the edges available to an attention graph and the experts activated for each token. This view connects local, dilated, and global attention with conditional expert computation without assuming that sparsity alone preserves document-level information. We specify normalized masked attention, distinguish denoising depth from sampling iterations, and derive graph and compute bounds. A reported sparse-expert configuration, denoted DualBudget, is analyzed across scientific summarization, multi-hop question answering, dialogue, and sentence-pair prediction. All supplied table measurements are retained. The length study shows gains over the listed baselines through 16K, together with measurable degradation as length increases. Ablations reveal mixed changes across the three reported scores, while the cost record indicates a 20% reduction relative to DiffuSeq under an unspecified timing convention. The analysis separates these observations from unmeasured claims about training speed, absorbing states, global coherence, and deployment memory.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.41487 [cs.CL] |
| (or arXiv:2610.41487v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41487 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Paul Raj [view email]
[v1]
Fri, 30 Oct 2026 09:43:20 UTC (1,418 KB)
来源：arXiv cs.CL · arxiv.org