arXiv cs.AI· Rui Fang, Hsi-Wen Chen, Chia-Ming Cheng, Ming-Syan Chen·· 3 天前AI 评分22
LoopQ：面向循环语言模型的量化方法
LoopQ: Quantization for Looped Language Models
AI 导读
针对循环语言模型（LoopLM）在训练后量化（PTQ）中因角色分布偏移、循环状态复用和递归误差累积导致的性能退化，研究者提出循环感知量化框架 LoopQ，在保留共享量化主干的同时引入轻量适配。在 W4A4 量化下，LoopQ 在四个主干上实现五任务平均准确率 67.8% 的平均相对提升，八个主干-数据集组合的平均困惑度相对下降保守估计为 75.2%。
正文 · 原文
Abstract:Looped language models (LoopLMs) increase effective computational depth by repeatedly applying shared Transformer blocks without increasing the parameter count. However, this reuse makes them particularly sensitive to post-training quantization (PTQ). We systematically study PTQ in LoopLMs and identify three sources of degradation: distribution shift across roles, state reuse across loop transitions, and recursive error accumulation. We propose LoopQ, a loop-aware PTQ framework that preserves a shared quantized backbone while introducing lightweight adaptations. LoopQ combines loop-aware activation scaling and selective transformations to accommodate changing hidden-state distributions, cross-loop state alignment to stabilize transitions, and trajectory-aware calibration to limit accumulated error. Under W4A4 quantization, LoopQ achieves a 67.8% average relative improvement in five-task mean accuracy across four backbones and a conservative estimate of 75.2% for the average relative perplexity reduction across eight backbone-dataset pairs, compared with the strongest static PTQ baseline for each comparison.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.16343 [cs.LG] |
| (or arXiv:2605.16343v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.16343 arXiv-issued DOI via DataCite |
Submission history
From: Rui Fang [view email]
[v1]
Fri, 8 May 2026 01:45:28 UTC (485 KB)
[v2]
Mon, 12 Oct 2026 16:51:06 UTC (242 KB)
来源：arXiv cs.AI · arxiv.org