arXiv cs.CL· Borja Aizpurua, Sukhbinder Singh, Rom\'an Or\'us·· 6 天前AI 评分29
权重聚类揭示大语言模型中相对排名的重要性
Weight Clustering Reveals the Importance of Relative Rank in Large Language Models
AI 导读
研究者对预训练 LLM 的权重矩阵做标量 K-means 聚类，仅用 16–64 个共享值替换原权重，在 Llama 3.1-8B-Instruct 和 SmolLM2-135M 上无需重训练即可保持较强准确率，仅微调聚类质心还能挽回 30–40% 的剩余精度损失。
正文 · 原文
Abstract:Large language models (LLMs) contain billions of parameters, but many exact weight values are not essential. We apply scalar $K$-means clustering to pretrained models, replacing each weight matrix with only 16--64 shared values. For Llama~3.1-8B-Instruct and SmolLM2-135M, this preserves strong accuracy without retraining, providing a simple way to compress LLMs on disk. Fine-tuning only the cluster centroids further recovers 30--40\% of the remaining accuracy loss. Using clustering as an analysis tool, we then keep the cluster assignments fixed and perturb only the shared values, to study which of their properties matter most. We find that preserving their \emph{relative rank} (which connections are stronger or weaker than others) is more important than preserving their exact magnitudes. Breaking this ordering greatly increases perplexity, while changing the values but keeping the ordering causes little loss in many middle and late layers. However, rank preservation alone is not sufficient. Model performance also depends on maintaining the variance of the reconstructed weights, while early layers remain more sensitive; perturbing the cluster assignments themselves shows that their spatial geometry is also load-bearing. We observe the same behavior across four LLMs from two architecture families. These results suggest that the ordering of weights, together with their overall variance, plays a central role in LLM compression and robustness.
| Comments: | 19 pages, 4 figures, 17 tables |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2603.17917 [cs.LG] |
| (or arXiv:2603.17917v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2603.17917 arXiv-issued DOI via DataCite |
Submission history
From: Borja Aizpurua Altuna [view email]
[v1]
Wed, 18 Mar 2026 16:55:13 UTC (167 KB)
[v2]
Fri, 9 Oct 2026 10:13:46 UTC (183 KB)
来源：arXiv cs.CL · arxiv.org