arXiv cs.CL· Ke Wan, Chen Chen·· 6 天前AI 评分24
RAMP：面向循环深度语言模型高效预填充的循环自适应混合精度方法
RAMP: Recurrent Adaptive Mixed Precision for Efficient Prefill in Recurrent-Depth Language Models
AI 导读
针对循环深度语言模型长上下文预填充中重复计算开销大的问题，研究者提出 RAMP（循环自适应混合精度），在循环深度早期用低精度权重与激活计算、末端恢复高精度。分析表明早期引入的量化误差会被更多后续循环步骤衰减，匹配预算实验也验证了这一预测。在最长 4K 上下文下，RAMP 保持接近全精度的预填充质量，在 NVIDIA H100 上实现最高 1.16x 端到端首次预填充加速，延迟降低 13.9%。
正文 · 原文
Abstract:Recurrent-depth language models improve inference-time reasoning by repeatedly applying a shared network to refine internal representations, but the additional recurrent computation also increases inference cost. This overhead is particularly pronounced during long-context prefill, where the full input sequence is repeatedly processed across recurrent depth. Weight-and-activation quantization offers a natural way to accelerate this computation through lower-precision arithmetic, yet repeatedly applying quantized computation introduces numerical perturbations that accumulate along the recurrent trajectory. We systematically analyze how such errors propagate across recurrent depth and show that quantization errors introduced earlier are attenuated by more subsequent recurrent steps and therefore have a smaller effect on the final representation. Matched-budget experiments strongly support this prediction, showing that placing quantized computation earlier in recurrent depth consistently yields the best fidelity. Based on this insight, we introduce RAMP (Recurrent Adaptive Mixed Precision), which performs early recurrent refinement using low-precision weight-and-activation computation and restores high precision for terminal refinement. Across context lengths up to 4K, RAMP preserves near-full-precision prefill quality while achieving up to 1.16x end-to-end first-prefill speedup on an NVIDIA H100, corresponding to a 13.9% latency reduction.
| Comments: | 16 pages, 3 figures, 6 tables. Preprint |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.13041 [cs.CL] |
| (or arXiv:2610.13041v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13041 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ke Wan [view email]
[v1]
Fri, 9 Oct 2026 05:57:33 UTC (95 KB)
来源：arXiv cs.CL · arxiv.org