arXiv cs.CL· Boyuan Zheng, Jing Xiong, Chenwei Wu, Hui Shen, Zesen Zhao, Hei Ting Una Chan, Xueshen Liu, Zhen Zhang, Liyue Shen, Zechun Liu, Z. Morley Mao·· 13 小时前AI 评分34
OnQ：用自生成轨迹在量化后恢复推理能力
Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories
AI 导读
研究团队提出低比特恢复流程 OnQ，结合同策略蒸馏与候选级 Dynamic KL，让量化学生模型在自己的前缀上接受全精度教师监督。在 Qwen3-1.7B、4B、8B 的 2-bit 和 3-bit 精度下，OnQ 取得九项基准最高平均分，2-bit 时每个模型规模、每项基准均超越所有基线，Qwen3-8B 平均分较最强基线提升 10.26 分。该方法还生成更短的回答，截断和重复错误更少。
正文 · 原文
Authors:Boyuan Zheng, Jing Xiong, Chenwei Wu, Hui Shen, Zesen Zhao, Hei Ting Una Chan, Xueshen Liu, Zhen Zhang, Liyue Shen, Zechun Liu, Z. Morley Mao
Abstract:Quantization-aware training (QAT) aims to recover reasoning capabilities in low-bit large language models (LLMs). Existing recipes typically supervise the quantized student with the original full-precision model on fixed response sequences. However, the resulting low-bit quantized models still struggle on long-horizon reasoning tasks and often fail to terminate within a token budget that suffices for the full-precision model. We identify two challenges in closing this gap. First, fixed-response supervision leaves a mismatch between training prefixes and those encountered during student generation, and this trajectory divergence grows as responses lengthen. Second, on its own prefixes, the student underpredicts some next-token candidates and overpredicts others, and this distributional misalignment becomes more frequent at high-entropy prefixes. We present OnQ, a low-bit recovery pipeline that combines on-policy distillation and candidate-wise Dynamic KL. Starting from block-wise calibration, the quantized student generates responses and receives full-precision teacher supervision on its own prefixes. Dynamic KL weights forward and reverse KL by each candidate's teacher-student probability ratio to balance correction of under- and overprediction. Across Qwen3-1.7B, 4B, and 8B at 2- and 3-bit precision, OnQ achieves the highest nine-benchmark average, with the largest gains on long-horizon reasoning benchmarks. At 2 bits, it outperforms every baseline on every benchmark at every model size, improving the Qwen3-8B average by 10.26 points over the strongest baseline. It also produces shorter responses with fewer truncations and repetitive errors.
| Comments: | 19 pages, 6 figures, 9 tables |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.29378 [cs.CL] |
| (or arXiv:2610.29378v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29378 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Boyuan Zheng [view email]
[v1]
Wed, 21 Oct 2026 08:03:24 UTC (177 KB)
来源：arXiv cs.CL · arxiv.org