arXiv cs.CL· Dongwei Wang, Jinhee Kim, Seokho Han, Denis Gudovskiy, Yohei Nakata, Tomoyuki Okuno, KhayTze Peong, Kang Eun Jeon, Jong Hwan Ko, Yiran Chen, Huanrui Yang·· 12 小时前AI 评分31
MoBiQuant：面向 Token 自适应任意精度 LLM 的混合比特量化框架
MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM
AI 导读
MoBiQuant 是一个任意精度的混合比特量化框架，按 token 敏感度动态调整权重精度，实现灵活的 LLM 推理。它提出多合一递归残差量化，可在运行时迭代重建更高精度权重，并用 token 感知路由器缓解精度相关的离群值迁移问题。实验显示其性能匹配或超越前沿单精度 PTQ，相比 SOTA 任意精度方法吞吐提升最高 1.34 倍，代码已开源。
正文 · 原文
Authors:Dongwei Wang, Jinhee Kim, Seokho Han, Denis Gudovskiy, Yohei Nakata, Tomoyuki Okuno, KhayTze Peong, Kang Eun Jeon, Jong Hwan Ko, Yiran Chen, Huanrui Yang
Abstract:Dynamic runtime latency and memory constraints necessitate flexible large language model (LLM) deployment, where an LLM can be inferred with various quantization precisions based on available computational resources. Recent work on such any-precision quantization either relies on hardware-inefficient vector quantization or induces additional scaling factors when switching between bit-widths. Meanwhile, existing post-training quantization (PTQ) methods calibrated for a fixed low precision show poor generalizability under runtime precision change. In this work, we attribute the source of poor generalization across bit-widths to a precision-dependent outlier migration phenomenon where the distribution of PTQ-sensitive tokens changes across precisions. Motivated by this observation, we propose MoBiQuant, a novel any-precision Mixture-of-Bits quantization framework that adjusts weight precision for flexible LLM inference based on token sensitivity. Specifically, we propose a many-in-one recursive residual quantization that can iteratively reconstruct higher-precision weights at runtime and mitigates outlier migration with a token-aware router to dynamically select the optimal inference precision of each this http URL experiments show that MoBiQuant matches or surpasses frontier single-precision PTQ while exhibiting strong elasticity, achieving significant memory savings and throughput gains of up to $1.34\times$ over state-of-the-art any-precision methods. Our code is available at this https URL.
| Comments: | NeurIPS 2026 Camera-ready |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2602.20191 [cs.LG] |
| (or arXiv:2602.20191v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.20191 arXiv-issued DOI via DataCite |
Submission history
From: Dongwei Wang [view email]
[v1]
Sat, 21 Feb 2026 21:11:08 UTC (3,462 KB)
[v2]
Mon, 25 May 2026 16:17:59 UTC (4,174 KB)
[v3]
Sat, 31 Oct 2026 19:11:45 UTC (5,528 KB)
来源：arXiv cs.CL · arxiv.org