arXiv cs.CL· Maximilian Idahl, Oleh Astappiev, Benedikt Droste, Robert Scholz, Bj\"orn Pl\"uster, Jan Philipp Harries, Robert J\"aschke, Wolfgang Nejdl·· 12 小时前AI 评分24
RefinedGAW：基于德国学术网 26 次抓取的德语学术网络数据集
RefinedGAW: A Refined German Academic Web Dataset
AI 导读
RefinedGAW 是从德国学术网 26 次半年度抓取中构建的德语数据集，共 1.22 亿份文档、3140 亿 token，并保留每段文本的完整抓取与采集历史。
正文
Abstract:The largest open German text datasets are built from broad web crawls, which provide scale but limited coverage of specific domains. We introduce RefinedGAW, a processed and annotated resource derived from 26 semi-annual crawls of the German Academic Web, totaling 122 million documents and 314 billion tokens, with full crawl and capture history of every text. Documents labeled as academic research account for 84% of German RefinedGAW tokens across HTML and PDF. At 87.6 billion tokens, this exceeds the 64.9 billion tokens in the German subsets of FineWeb2 and FinePDFs, despite their almost five-times-larger combined size. RefinedGAW also contains substantial amounts of educational, math-heavy, and expert-oriented content with minimal commercial bias. Across HTML and PDF, 88% of German RefinedGAW documents have no near-duplicate candidate match in the three evaluated reference datasets, supporting complementary coverage. We release a queryable metadata index for fine-grained data selection and a rehydration tool for retrieving and processing source documents.
| Comments: | 26 pages, 13 figures, 6 tables. Accepted at the 13th Web as Corpus Workshop (WaC-13), 2026 (non-archival) |
| Subjects: | Computation and Language (cs.CL); Information Retrieval (cs.IR) |
| Cite as: | arXiv:2610.34572 [cs.CL] |
| (or arXiv:2610.34572v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34572 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Maximilian Idahl [view email]
[v1]
Sun, 25 Oct 2026 21:14:01 UTC (470 KB)
来源：arXiv cs.CL · arxiv.org