arXiv cs.AI· Sai Sreenivas Chintha, Chidera Biringa, Rishab Sharma, Anusha Agrawal, Prachi Badarayani, Gemma Zhang, Nigel Boachie Kumankumah, Marco Ayala, Sulaiman Vesal·· 11 小时前AI 评分31
CADENCE：让模型学习一次解码多远，生成吞吐最高提升 1.3 倍
Next Token or Vector? Learning How Far to Decode
AI 导读
研究者提出 CADENCE（Context-Adaptive DEcoding）生成方法，让模型同时学习草拟什么内容和草拟多长。该方法用高斯预测头将未来片段表示为连续隐向量，由共享解码器并行转成 token 预测，并通过学习到的路由器根据模型自身隐状态选择预测长度，以单 token 生成作为回退。
正文 · 原文
Abstract:Standard autoregressive decoding advances one token at a time, even when several upcoming tokens can be predicted together. We propose CADENCE (Context-Adaptive DEcoding), a generation method that learns both what to draft and how far to draft. Gaussian prediction heads represent future spans as continuous latent vectors, and a shared decoder converts the selected vector into token predictions in parallel. A learned router selects the prediction length from the model's own hidden states and predicted Gaussian parameters, adapting generation granularity to the current context with single-token generation as a fallback. The backbone verifies each proposed continuation and retains the prefix that matches its greedy predictions. Experiments with Llama-3.1-8B-Instruct on Alpaca, GSM8K, and HumanEval show that router-selected positions yield higher draft acceptance than applying the corresponding prediction length everywhere. CADENCE achieves up to 1.3x higher generation throughput than standard greedy decoding.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Emerging Technologies (cs.ET) |
| Cite as: | arXiv:2610.22788 [cs.LG] |
| (or arXiv:2610.22788v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22788 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sai Sreenivas Chintha [view email]
[v1]
Fri, 16 Oct 2026 01:00:34 UTC (1,164 KB)
来源：arXiv cs.AI · arxiv.org