arXiv cs.CL· Iffat Maab, Waleed Jamil, Raphael Schmitt·· 11 小时前AI 评分22
DunbaaBERT：从牺牲到语义的乌尔都语 RoBERTa 模型
DunbaaBERT: From Sacrifice to Semantics
AI 导读
研究团队推出 DunbaaBERT，一组从零训练的乌尔都语 RoBERTa-base 模型，使用 32k、52k、96k 三种 Byte-BPE 词表，训练语料为去重后的 17 GB 乌尔都语数据。
正文 · 原文
Abstract:Large language models have achieved strong performance across many NLP tasks, yet Urdu remains comparatively underexplored due to limited resources and fragmented evaluation settings. To address this gap, we introduce DunbaaBERT, a family of Urdu RoBERTa-base models trained from scratch with Byte-BPE vocabularies of 32k, 52k, and 96k tokens on a deduplicated 17 GB Urdu corpus. We evaluate DunbaaBERT across intrinsic and downstream Urdu NLP benchmarks covering linguistic acceptability, news classification, offensive language detection, and sentiment analysis, while analyzing vocabulary-size effects on performance, pre-training dynamics, and computational efficiency. Across benchmarks, the DunbaaBERT variants achieve competitive performance against strong multilingual baselines while consistently maintaining favorable efficiency trade-offs. Under our controlled experimental setup, increasing the vocabulary size does not consistently improve downstream effectiveness, with DunbaaBERT$_{32k}$ repeatedly providing the strongest overall efficiency profile. Overall, our results demonstrate that carefully curated Urdu-specific encoder models can remain highly competitive despite comparatively compact model and training scales. All models are released under the MIT license.
| Comments: | Camera-ready version. Version of record: this https URL |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.26935 [cs.CL] |
| (or arXiv:2605.26935v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.26935 arXiv-issued DOI via DataCite |
|
| Journal reference: | Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026, pp. 2074-2093 |
Submission history
From: Raphael Schmitt [view email]
[v1]
Tue, 26 May 2026 12:28:20 UTC (1,207 KB)
[v2]
Fri, 6 Nov 2026 16:57:27 UTC (1,208 KB)
来源：arXiv cs.CL · arxiv.org