arXiv cs.AI· Bowen Zheng, Weijian Luo, Guang Yang, Colin Zhang, Tianyang Hu·· 11 小时前AI 评分32
用可变码本大小量化驯服熵悬崖：面向自回归视觉生成的 VCQ 方法
Taming the Entropy Cliff: Variable Codebook Size Quantization for Autoregressive Visual Generation
AI 导读
研究者提出可变码本大小量化（VCQ），让码本大小沿序列从 K_min=2 单调增长到 K_max，以应对自回归视觉生成中的"熵悬崖"现象——在 ImageNet 上 K=16384 时，仅约 256 个位置中的前 2 个之后条件分布就近乎确定。
正文 · 原文
Abstract:Most discrete visual tokenizers rely on a default design: every position in the sequence shares the same codebook. Such a constant-codebook design can consume the finite-data entropy budget very quickly. We observe that the per-position empirical conditional entropy decays so quickly along the sequence that, after a few positions, the conditional distribution becomes essentially deterministic. On ImageNet with $K=16384$, this happens after only about 2 out of 256 positions, when most prefixes already identify individual images in the dataset. We call this phenomenon the Entropy Cliff. Under full conditional utilization, larger codebooks exhaust the finite-data entropy budget in fewer positions. The empirical entropy decreases more gradually in our language comparison, where the effective entropy per position stays below the codebook capacity. To address this, we propose Variable Codebook Size Quantization (VCQ), where the codebook size $K_t$ grows monotonically along the sequence from $K_{\min}=2$ to $K_{\max}$, leaving the loss function, parameter count, and AR training procedure unchanged. With a vanilla autoregressive Transformer and standard next-token prediction, VCQ-Base achieves gFID w/o CFG 14.80 on ImageNet $256\times256$, compared with 27.98 and 18.16 for constant codebooks of size 16384 and 8192, respectively. Scaled up, it reaches gFID 1.71 with 684M autoregressive parameters, without any extra training techniques such as semantic regularization or causal alignment. The extreme information bottleneck at $K_{\min}=2$ encourages semantic concentration in early encoder features: a linear probe on features from only the first 10 positions reaches 43.8% top-1 accuracy on ImageNet, compared to 27.1% for uniform codebooks. These results show that what matters is not only the total capacity of the codebook, but also how that capacity is distributed and organized.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.06207 [cs.CV] |
| (or arXiv:2605.06207v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2605.06207 arXiv-issued DOI via DataCite |
Submission history
From: Bowen Zheng [view email]
[v1]
Thu, 7 May 2026 13:13:30 UTC (26,460 KB)
[v2]
Thu, 5 Nov 2026 08:11:43 UTC (26,457 KB)
来源：arXiv cs.AI · arxiv.org