arXiv cs.CL· Malte Ostendorff·· 6 天前AI 评分24
面向 LLM 预训练数据的 Host-Aware 网页抓取策略研究
Host-Aware Web Crawling for LLM Pretraining Data
AI 导读
研究发现，文档质量分类器得分约一半的差异来自不同 host 之间，且同一 host 少量页面即可预估其余页面质量。模拟显示，用中心性指标选择抓取哪些 host 相比随机排序收益有限，而先采样少量页面再对高分 host 加深抓取，每次抓取获得的高质量页面是均匀深度或中心性分配的数倍。
正文 · 原文
Abstract:Web crawling is a major source of large language model (LLM) pretraining data, yet data-selection pipelines discard a large share of crawled pages. We investigate how a crawler should allocate its budget across and within hosts to collect more high-quality pretraining data. Combining document-level quality classifier scores, web-graph centrality metrics, and Common Crawl's per-host crawl coverage, we find that roughly half of the variation in classifier scores occurs between hosts. In the crawls we analyze, a few pages from a host also provide a useful estimate of the quality of its remaining pages. Common Crawl uses harmonic centrality, a graph centrality metric, to guide its crawl allocation. In our simulations, using centrality to choose which hosts to crawl yields limited gains over random ordering and performs worse for some classifiers. For deciding how deeply to crawl each host, sampling a few pages and then fetching more from hosts with higher-scoring samples collects several times more high-quality pages per fetch than uniform depth or centrality-based allocation. Finally, Common Crawl's observed per-host coverage has little correlation with the evaluated quality signals. We estimate that tens of millions of high-quality pages remain among known but uncrawled URLs in each crawl examined, with most concentrated on a small share of hosts.
| Comments: | Accepted at WaC-13 @ EMNLP 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.13321 [cs.CL] |
| (or arXiv:2610.13321v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13321 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Malte Ostendorff [view email]
[v1]
Fri, 9 Oct 2026 10:03:04 UTC (163 KB)
来源：arXiv cs.CL · arxiv.org