arXiv cs.CL· Qingjie Zhang, Di Wang, Haoting Qian, Liu Yan, Tianwei Zhang, Ke Xu, Qi Li, Minlie Huang, Hewu Li, Han Qiu·· 12 小时前AI 评分38
研究通过 token 推测 LLM 中文训练数据污染：GPT 词表超 40% 长中文 token 涉色情或赌博
Speculating LLMs' Chinese Training Data Pollution from Their Tokens
AI 导读
一项 arXiv 研究通过分析 token 词表来推测 LLM 的中文训练数据污染，发现 GPT 词表中超过 40% 的长中文 token（含两个以上汉字）与色情或在线赌博内容相关，在 GPT 及另外 23 个 LLM 的实验中表现最差。
正文 · 原文
Abstract:Tokens are basic elements in the datasets for LLM training. It is well-known that many tokens representing Chinese phrases in the vocabulary of GPT (4o/4o-mini/o1/o3/4.5/4.1/o4-mini) are indicating contents like pornography or online gambling. Based on this observation, our goal is to locate Polluted Chinese (PoC) tokens in LLMs and study the relationship between PoC tokens' existence and training data. (1) We give a formal definition and taxonomy of PoC tokens based on the GPT's vocabulary. (2) We build a PoC token detector via fine-tuning an LLM to label PoC tokens in vocabularies by considering each token's both semantics and related contents from the search engines. (3) We study the speculation on the training data pollution via PoC tokens' appearances (token ID). Experiments on GPT and other 23 LLMs indicate that tokens widely exist while GPT's vocabulary behaves the worst: more than 40% long Chinese tokens (i.e., a token with more than two Chinese characters) are either porn or online gambling. We validate the accuracy of our speculation method on famous pre-training datasets like C4 and Pile. Then, considering GPT-4o, we speculate that the ratio of "Yui Hatano" related webpages in GPT-4o's training data is around 0.5%.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2508.17771 [cs.CL] |
| (or arXiv:2508.17771v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2508.17771 arXiv-issued DOI via DataCite |
Submission history
From: Qingjie Zhang [view email]
[v1]
Mon, 25 Aug 2025 08:08:51 UTC (12,732 KB)
[v2]
Tue, 30 Sep 2025 07:36:58 UTC (12,733 KB)
[v3]
Fri, 16 Oct 2026 03:49:44 UTC (12,730 KB)
来源：arXiv cs.CL · arxiv.org