arXiv cs.CL· Houssam Eddine Boukhalfa, Mourad Bouache, Mourad Dahmane·· 13 小时前AI 评分31
AQL-Lite：通过以数据为中心的持续预训练为阿拉伯语 LLM 实现高效知识获取
AQL-Lite: Enabling Efficient Knowledge Acquisition for Arabic LLMs via Data-Centric Continual Pretraining
AI 导读
AQL-Lite 框架通过质量优先的数据筛选流程与部分参数持续预训练，将多语言 LLM 适配到现代标准阿拉伯语。在 1B token 预算下，AQL 精选数据使四项阿拉伯语基准平均分比原始网络文本提升 3.51 分；5B token 对比中，AQL-Lite 以 52% 的实际耗时恢复全参数 CPT 94% 的增益（平均 73.19 对 73.58）。
正文 · 原文
Abstract:Training Arabic large language models faces challenges from noisy web data, dialectal variation, and the high cost of continual pretraining (CPT). We introduce AQL-Lite (AQL: Arabic Quality Language), a time- and resource-efficient framework for adapting a multilingual LLM to Modern Standard Arabic (MSA). AQL-Lite couples a quality-first curation pipeline (structural cleaning, heuristic and model-based filtering, MSA selection, and deduplication) with a partial-parameter CPT recipe that trains RS-LoRA adapters across the network together with the last eight transformer blocks and the embedding matrix, under layer-wise learning rates. At a matched 1B-token budget, CPT on AQL-curated data raises a four-benchmark Arabic mean by 3.51 points over CPT on raw web text, which yields no gain over the base model. In a controlled 5B-token comparison, AQL-Lite recovers 94% of the gain of full-parameter CPT (mean 73.19 vs. 73.58) in 52% of the wall-clock time and outperforms adapter-only baselines on every benchmark, and a component ablation shows that adapters, unfrozen blocks, and embedding training each contribute. Scaled to 40B MSA tokens and instruction-tuned on one million examples, the resulting 12B model scores 75.3 on the seven-benchmark Open Arabic LLM Leaderboard v2 (OALL v2) suite. These results suggest that partial-parameter CPT on a carefully curated corpus is a practical route to efficient MSA adaptation.
| Comments: | 15 pages, 7 figures, 10 tables |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.30344 [cs.CL] |
| (or arXiv:2610.30344v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.30344 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Houssam Eddine Boukhalfa [view email]
[v1]
Wed, 21 Oct 2026 19:04:28 UTC (930 KB)
来源：arXiv cs.CL · arxiv.org