arXiv cs.CL· Cheng Luo, Xinnan Dai, Zhongzhu Zhou, Zefan Cai, Ruijie Zhu, Tianyu Zhang, Junjie Hu·· 2 天前AI 评分32
onóna:双嵌入表与绑定解嵌出参数化方法
on\'ona: Twin Embeddings with Tied Unembeddings
AI 导读
onóna 提出一种双表参数化方法,用两张嵌入表的平均值作为 Transformer 输入,输出头则融合这两张表各自定义的 softmax 分布,混合权重由仅依赖当前输入 token 的门控决定,且可在推理前预计算。
正文 · 原文
Abstract:Weight tying reduces the parameter cost of language models, but the usual tied output head is limited to a single softmax. We introduce onóna, a two-table parameterization that increases output expressiveness while preserving input-output weight sharing. The Transformer takes the average of two embedding tables as input, and the output head combines the softmax distributions defined by those same tables. A gate conditioned only on the current input token determines the mixture weight. Its values can be precomputed for inference, and the Transformer body remains unchanged. We evaluate onóna across Dense, Gated DeltaNet hybrid, mixture-of-experts, and Attention Residual architectures at 142M, 330M, and 1B scales. Under matched token budgets, onóna lowers final validation loss and external perplexity in 11 of 12 comparisons and raises the average accuracy over eight downstream tasks in 10 of 12. The one exception on loss is the 1B MoE model. The cost is one extra embedding table and one extra vocabulary projection.
| Comments: | 18 pages |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.10556 [cs.CL] |
| (or arXiv:2611.10556v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.10556 arXiv-issued DOI via DataCite |
Submission history
From: Cheng Luo [view email]
[v1]
Sat, 7 Nov 2026 07:55:25 UTC (79 KB)
[v2]
Fri, 13 Nov 2026 01:06:47 UTC (79 KB)
来源:arXiv cs.CL · arxiv.org