arXiv cs.AI· Qiyan Zhao, Xiaofeng Zhang, Xu-Yao Zhang, Shijie Han, Zhepeng Li, Shunzhi Zhu, Da-Han Wang·· 12 小时前AI 评分31
CoTA++：从信息流视角理解 dMLLM 中的重复诅咒
CoTA++: Understanding the Repeat Curse in dMLLMs from an Information Flow Perspective
AI 导读
针对扩散多模态大模型（dMLLM）近似缓存导致的“重复诅咒”，研究者提出免训练、即插即用的框架 CoTA++，从步内、跨步、跨层三处信息流断裂入手修复。在三个 dMLLM 和两种缓存后端上，CoTA++ 将长回答的样本重复率从最高 99.6% 降至最多 17.0%，并在九个基准上找回大部分因缓存损失的准确率，额外开销低于 15%。
正文 · 原文
Abstract:Diffusion-based multimodal large language models (dMLLMs) decode by iterative denoising with bidirectional attention, which precludes the standard autoregressive key--value cache. Practical deployments therefore rely on approximate caching that reuses intermediate states across denoising steps. Such caching, however, often makes a response repeat one token at adjacent positions in a run that breaks the sentence, a failure we term the \emph{Repeat Curse}. Tracing the context-token information flow reveals three disruptions. Within a step, caching disperses the attention that anchors a position on nearby context tokens. Across steps, similarity-based refresh is not staleness-aware, so some positions are decoded against stale key--value states. Across layers, the deep-layer entropy of the context fails to converge where repetition arises. We accordingly propose CoTA++, a training-free, plug-and-play framework that counteracts each disruption where it originates: Context-Token Attention Re-anchoring re-forms the attention routing against the current context, Decode-Aware Refresh reserves part of the cache's refresh budget for the positions about to be committed, and Context-Token Entropy-Guided Voting discounts the confidence of a position by the deep-layer entropy of its context, which reorders the unmasking. The three components act at non-overlapping stages of a decoding step and compose freely with existing dMLLMs and caches. Across three dMLLMs and two caching backends, CoTA++ reduces the sample repetition rate of long responses from up to 99.6\% to at most 17.0\%, recovers most of the accuracy lost to caching on nine benchmarks, and keeps the speedup of caching at under 15\% overhead, making approximate caching reliable for dMLLMs.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.23290 [cs.AI] |
| (or arXiv:2610.23290v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.23290 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Qiyan Zhao [view email]
[v1]
Fri, 16 Oct 2026 10:58:03 UTC (2,876 KB)
来源：arXiv cs.AI · arxiv.org