跳到正文
arXiv cs.CL· Stephen Chung, Wenyu Du·· 10 小时前AI 评分29

Chorus:共享 N-gram 嵌入表配合逐层适配

Chorus: Shared N-gram Embeddings with Layer-wise Adaptation

AI 导读

Chorus 提出一套共享 N-gram 嵌入表服务整个网络,每层通过自身投影层并按当前隐藏状态计算缩放因子来读取,以 1.2B 表参数取得优于 FactorEngram 和 DeepSeek Engram 9.6B 参数的文本预测效果,表体积缩小 87%。

正文 · 原文

View PDF HTML (experimental)

Abstract:Language models can gain capacity at little computational cost from n-gram embedding tables: large tables of learned vectors for short sequences of consecutive tokens, which are looked up and added to the model's input or to its intermediate layers. Recent designs commonly use a separate n-gram embedding table for each layer, which makes the tables costly in memory and prevents layers from sharing what they learn. However, a single shared table gives every layer the same embedding, leaves no room for layer-wise preferences, and performs poorly in practice. We propose Chorus, in which one set of n-gram embedding tables serves the whole network and every layer reads it in its own way. Each layer maps the refined embedding through its own projection layer, after rescaling its components by factors computed from its current hidden state. Chorus matches the quality of existing n-gram methods with 87% smaller tables: with 1.2B table parameters it predicts text better than FactorEngram and DeepSeek's Engram do with 9.6B. It outperforms Engram, TN-gram and FactorEngram in every setting where we compared them, across table sizes from 20M to 9.6B parameters. We also show that the recently proposed memory-efficient optimizer Ember works well with Chorus, saving 99.2% of table-optimizer memory compared with AdamW without degrading training performance. These results enable further scaling of n-gram embedding tables along the table-size axis by reducing memory use and allowing shared knowledge to be reused across layers.
Comments: 18 pages. Code: this https URL. Part of the Autonomous Research Program: this https URL
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2611.23386 [cs.CL]
  (or arXiv:2611.23386v2 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.23386

arXiv-issued DOI via DataCite

Submission history

From: Stephen Chung [view email]
[v1] Tue, 17 Nov 2026 19:00:05 UTC (181 KB)
[v2] Thu, 19 Nov 2026 12:49:23 UTC (181 KB)

来源:arXiv cs.CL · arxiv.org