arXiv cs.CL· Sai Kiran Bathula·· 12 小时前AI 评分38
冻结 16 MB 词表能否替代可训练嵌入？小型与三值语言模型实验
Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models
AI 导读
研究者用一个由预训练嵌入经主成分投影和符号二值化得到的 512-bit 冻结码表（16 MB、无可训练参数）替换 Gemma 3 270M 的嵌入表，并在相同 2.05B tokens 上从头训练同构模型对比。
正文 · 原文
Abstract:In a small language model the vocabulary is the single largest part. In Gemma 3 270M the embedding table holds 168M of 268M parameters, and because quantisation methods leave the embedding in 16-bit, it is also most of the file once the rest of the model is compressed. We ask what that table actually buys. We replace it with one frozen table of 512-bit codes, 16 MB, with no trainable parameters, derived once from a pretrained embedding by principal-component projection and sign binarisation, and we train the same Gemma-shaped body from scratch on the same 2.05B tokens with and without it. The learned table buys 0.20 nats of validation loss for 2.6 times the trainable parameters, but on LAMBADA, HellaSwag and PIQA the frozen-table model is better or tied; only ARC-Easy favours the learned table. With a ternary BitNet b1.58 body the whole model is a 46 MB file, still ties the float model on LAMBADA and PIQA, and on a laptop CPU generates 615 to 740 tokens per second in 62 MB of memory, five to six times the speed of the float model's 8-bit GGUF in this http URL at a seventh of the memory. The frozen table wins by 1.7 to 2.3 nats on rare words, accepts 10,000 new words after training with a 3% cost where the learned table collapses by 43%, and all of it is released with the weights, the data and every evaluation answer.
| Comments: | 9 pages, 2 figures, 6 tables. Code, data, checkpoints and per-question results: this https URL. Preprint DOI: https://doi.org/10.5281/zenodo.23382634 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.22428 [cs.CL] |
| (or arXiv:2610.22428v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22428 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Saikiran Bathula [view email]
[v1]
Thu, 15 Oct 2026 06:34:18 UTC (80 KB)
来源：arXiv cs.CL · arxiv.org