arXiv cs.AI· Mohammed Nayeem Khan, Logesh L, Aliyas Ahmad Misri·· 10 小时前AI 评分24
Qwen3-0.6B 按需精度：不确定解码步升级权重位宽
Precision on Demand: Escalating Weight Bits at Uncertain Decoding Steps in a Small Local Model
AI 导读
研究在 Qwen3-0.6B 上以 8-bit 存储权重、默认用高 4 位解码，仅在低精度输出分布不确定时按 8-bit 重算该步，成本按每 token 读取的权重字节计。
正文 · 原文
Abstract:Local decoding usually fixes one weight precision for the whole generation. We study a different allocation: store 8-bit weights, decode with their top 4 bits, and redo a step at 8 bits only when the low-precision output distribution looks uncertain (entropy above a threshold or a small top-two margin). Quantization is simulated, the model is Qwen3-0.6B, and the benchmark is GSM8K. Cost is counted in weight bytes read per token. On 36724 replayed steps the entropy of the low pass ranks argmax flips with AUC 0.93. At a 15% escalation budget it catches 78% of flips and the margin gate 94%, against the 15% expected for a random gate of equal rate (a computed expectation, not a run). In 150 live problems all-high scores 64.0% and all-low 55.3% (McNemar p=0.053). An entropy gate with a 16% escalation rate scores 62.7% at 0.87 of the all-high weight bytes. Its paired difference is -1.3 (-8.0 to +5.3) points to all-high and +7.3 (+0.0 to +14.7) points to all-low (p=0.080), so the live data do not separate the gate from either end. Verified speculation reaches zero KL at similar bytes in the same cost model. A live random control, a thinking-mode phase rule, a second model and live KL were not run.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.14418 [cs.AI] |
| (or arXiv:2611.14418v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14418 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Logesh L [view email]
[v1]
Tue, 10 Nov 2026 17:57:51 UTC (901 KB)
来源：arXiv cs.AI · arxiv.org