arXiv cs.CL· Quang Anh Kiet Tran, Nam Anh Tran, Manh Hung Trinh, Thanh Danh Le, Khang Nhat Hoang Vo, Huynh Dai Nhan Tran·· 13 小时前AI 评分26
GoiBERT：两阶段知识蒸馏打造的紧凑越南语语言模型
GoiBERT: Compact Vietnamese Language Models via Two-Stage Knowledge Distillation
AI 导读
GoiBERT 推出 26M 和 49M 参数的两个紧凑越南语编码器，直接处理原始文本、无需外部分词。
正文 · 原文
Abstract:Vietnamese pre-trained encoders have improved rapidly, but the strongest ones are large, multilingual-vocabulary models or depend on external word segmentation, which makes them costly to deploy. We present GoiBERT, a family of two compact Vietnamese encoders with 26M and 49M parameters that operate directly on raw text. GoiBERT learns from two complementary teachers. A same-tokenizer Vietnamese encoder first teaches general-purpose representations. A much larger teacher then provides task-level guidance through soft labels that are generated without ever labeling its own training data. Across eight benchmarks covering text classification, aspect-based sentiment analysis, natural language inference, and named entity recognition, GoiBERT-large achieves the highest average score among non-teacher models. It does so with fewer than half the parameters of the strongest compact baseline and twice its throughput. GoiBERT-base retains 99% of that baseline's performance at a quarter of its size. Our analysis shows that compression is nearly lossless for sentiment and classification tasks but remains costly for inference and entity recognition, which points to clear directions for future work. Our code and models are available at: this https URL
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.30067 [cs.CL] |
| (or arXiv:2610.30067v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.30067 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Khang Vo Hoang Nhat [view email]
[v1]
Wed, 21 Oct 2026 17:24:36 UTC (182 KB)
来源：arXiv cs.CL · arxiv.org