arXiv cs.CL· Parikshit Bansal, Sujay Sanghavi·· 11 小时前AI 评分32
TTCD：面向语言建模的 token 时间连续扩散模型
Token Time Continuous Diffusion for Language Modeling
AI 导读
研究者提出 token 时间连续扩散模型 TTCD，在连续空间中确定性地将高斯噪声映射为最终 token 画布，无需额外采样，并引入逐 token 时间概念，让部分 token 更快从噪声转为 token。在 OpenWebText 上训练的 160M 参数 TTCD 经自蒸馏后，高加速比下无条件生成质量与同类模型相当，条件生成和数独求解表现更优。该工作已被 NeurIPS 2026 接收。
正文 · 原文
Abstract:In this paper we introduce token time continuous diffusion (TTCD), a new diffusion language model which (a) operates in continuous space, deterministically mapping Gaussian noise to a final token canvas with no further sampling, and crucially (b) incorporates a new notion of per-token times, with some tokens proceeding from noise to token at a faster rate than others. Continuous space modeling helps TTCD avoid the parallel sampling of multiple tokens, which is a key source of inaccuracy at high speedups for models that iterate purely in discrete space. The notion of per-token times helps TTCD to better model conditional generation, allows for more sure tokens to proceed at a faster rate, and allows for differentiated inter-token influences during refinement. TTCD outperforms discrete models at high speedups. We train a 160M parameter TTCD model on OpenWebText, and then self-distill it; we find that at high speedups we are comparable in unconditional generation quality, and outperform in conditional generation, several existing models of similar size trained, on the same data, and self-distilled. We achieve similar gains in Sudoku solving as well.
| Comments: | Accepted to NeurIPS 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.14106 [cs.CL] |
| (or arXiv:2607.14106v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.14106 arXiv-issued DOI via DataCite |
Submission history
From: Parikshit Bansal [view email]
[v1]
Thu, 7 May 2026 16:23:06 UTC (659 KB)
[v2]
Wed, 4 Nov 2026 16:32:30 UTC (1,121 KB)
来源：arXiv cs.CL · arxiv.org