arXiv cs.AI· Vinko Sabol\v{c}ec, Bettina Messmer, Yassine Turki, Martin Jaggi·· 9 天前AI 评分22
将英语质量分类器适配于多语言 LLM 预训练数据筛选
Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection
AI 导读
一项研究提出多语言适配方法，将现有英语质量分类器扩展至 100 多种语言，用于 LLM 预训练数据筛选。该方法在 Transformer encoder-only 模型嵌入之上训练小型多层感知机，以英语分类器对机器翻译文本的打分为标签。1B、3B 和 8B 规模实验显示，其下游 LLM benchmark 表现与现有多语言基于模型的筛选基线相当，且未损害区域与文化知识基准。
正文 · 原文
Abstract:Recent advances in large language model (LLM) pretraining highlight the role of high-quality training data in improving performance. While model-based filtering has proven effective in selecting high-quality subsets from web-scale corpora, especially for high-resource languages, low-resource languages face challenges due to limited availability of annotated data. This work explores extending quality filtering to over 100 languages by proposing a multilingual adaptation approach that converts an existing English quality classifier into a multilingual variant. Our approach proposes training a small multi-layer perceptron on top of Transformer encoder-only model embeddings, using multilingual text as input and scores obtained from English classifiers applied to machine-translated text as labels. Our 1B, 3B and 8B scale experiments show that our approach maintains the downstream LLM benchmark performance of existing multilingual model-based filtering baselines, without harming regional and cultural knowledge benchmarks. To further evaluate cross-lingual generalization, we compare classifier scores of high-quality synthetic data and web samples, and the correlation of classifier scores with LLM-based ones, revealing that the classifier can learn the scoring criteria of its original English variant, even for languages not included in its training data.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.11585 [cs.CL] |
| (or arXiv:2610.11585v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11585 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Vinko Sabolčec [view email]
[v1]
Thu, 8 Oct 2026 09:33:32 UTC (1,170 KB)
来源：arXiv cs.AI · arxiv.org