arXiv cs.CL· Mich-Seth Owusu·· 12 小时前AI 评分36
Africa-Papers：面向非洲基础模型的 100 亿 Token 多领域学术语料库
Africa-Papers: A 10-Billion-Token Multi-Domain Academic Corpus for African Foundation Language Models
AI 导读
研究团队发布 Africa-Papers，这是迄今最大的泛非多领域学术预训练语料库，含 844,807 篇全文论文、10.15 billion tokens，分布于 879 个 Parquet 分片。
正文 · 原文
Abstract:Large Language Models (LLMs) depend fundamentally on the diversity, depth, and factual rigor of their pre-training corpora. However, contemporary foundational pre-training datasets (such as The Pile, S2ORC, and Dolma) exhibit a severe structural omission: African scientific, medical, legal, and agricultural scholarship constitutes less than 0.5% of global pre-training tokens. Consequently, state-of-the-art models suffer from systematic failure modes when reasoning about tropical epidemiology, localized smallholder agro-ecologies, regional economic governance, and indigenous linguistic structures. To resolve this representation deficit and advance African intellectual sovereignty in artificial intelligence, we introduce Africa-Papers, the largest curated multi-domain Pan-African academic pre-training corpus to date. Comprising over 844,807 full-text papers spanning 10.15 billion tokens across 879 Parquet shards, Africa-Papers unifies four primary scientific streams: (1) continental peer-reviewed journals indexed in DOAJ across 31 African nations, (2) the CGIAR CGSpace agricultural repository, (3) African-authored and African-focused STEM/AI preprints on arXiv, and (4) open-access international publications by scholars affiliated with African institutions indexed via OpenAlex. The entire corpus is processed into layout-aware, clean Markdown preserving section hierarchies, mathematical equations, and tabular data, accompanied by exact BPE token counts and a standardized 8-domain taxonomy. We release Africa-Papers openly on the Hugging Face Hub under the AfriSpeech organization to accelerate continual pre-training and domain evaluation for foundation models globally.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.28173 [cs.CL] |
| (or arXiv:2610.28173v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.28173 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mich-Seth Owusu [view email]
[v1]
Tue, 20 Oct 2026 13:14:58 UTC (11 KB)
来源：arXiv cs.CL · arxiv.org