arXiv cs.AI· Haowen Shi, Xuzhe Yan, Fahao Chen, Peng Li·· 14 小时前AI 评分31
基于预填充的解码长度预测存在根本性极限：LLM 推理调度预测器受自回归随机性约束
On the Fundamental Limits of Prefill-Based Decoding Length Prediction
AI 导读
研究揭示当前 LLM 解码长度预测基准将预测器质量与任务难度混为一谈，模型、数据集或采样配置一变，准确率即崩塌。作者将解码视为停止随机过程，推导出每个预填充侧方法都必须遵守的误差界，并给出方差仿射定律 σ=αμ+b。实验显示 MLP、EGTP、FiLM、soft-label 与 MoE 等预测器在固定条件下性能趋同并逼近该界，证明系统误差主导模型误差。
正文 · 原文
Abstract:Predicting how many tokens a large language model (LLM) will decode, using only prefill-phase hidden states, is a key primitive for inference scheduling. We show that current benchmarks conflate predictor quality with task hardness: when the target model, dataset, or sampling configuration changes, reported accuracy collapses even though the predictor is unchanged. We trace this instability to an overlooked error source---the inherent stochasticity of autoregressive decoding---and derive bounds that every prefill-side method must obey. Our analysis treats decoding as a stopped stochastic process. The per-step stopping probability of a response is not constant but ramps up over hundreds of steps, so the decode length inherits a central limit theorem from the many local decisions that determine where a response ends; this yields per-prompt normality with an explicit error term, together with an affine variance law $\sigma=\alpha\mu+b$ whose slope is the coefficient of variation of the number of response segments. Empirically, the normal law reproduces the per-prompt MAE and the central mass of the decode-length distribution to within a few percent, and the residual departures from normality (mild right skew, slight excess kurtosis) are the ones predicted by the sampling-noise correction. Under fixed model, data, and sampling conditions, a broad design space of predictors (MLP, EGTP, FiLM, soft-label, and Mixture-of-Experts variants) converges to similar performance close to the bound, confirming that \emph{system error dominates model error}. We argue for a benchmarking protocol that fixes the model, data, and sampling parameters and that reports relative-error metrics alongside absolute ones. Code and analysis tools are available at this https URL.
| Comments: | 26 pages, 4 figures |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Information Theory (cs.IT) |
| Cite as: | arXiv:2610.40541 [cs.AI] |
| (or arXiv:2610.40541v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.40541 arXiv-issued DOI via DataCite |
Submission history
From: Haowen Shi [view email]
[v1]
Thu, 29 Oct 2026 17:36:01 UTC (542 KB)
来源：arXiv cs.AI · arxiv.org