arXiv cs.CL· Anastasia Voznyuk, Andrey Grabovoy·· 6 天前AI 评分22
机器生成文本检测器中的伪影依赖问题研究
Measuring Artifact Reliance in Machine-Generated Text Detectors
AI 导读
一项被 EMNLP 2026 DocInsights workshop 接收的研究发现，机器生成文本检测器在科学文献基准上的高分部分源于 PDF 提取伪影：人类文本保留原始换行，生成文本被提取为单一文本块。
正文 · 原文
Abstract:Detectors of machine-generated text score almost perfectly on scientific-literature benchmarks, implying we can reliably distinguish with these detectors human and machine-generated documents. We looked at ten scientific-document subsets from six popular resources, and on several of them that apparent ease is a side effect of how the documents were put together. A logistic regression using 27 superficial features reached 0.833-0.997 macro-F1, and on the three arXiv subsets it outperformed every detector we evaluate with F1-scores 0.951-0.997. The strongest cue to the detector is the PDF extraction artifact: human texts retain their original line breaks, whereas generated texts are extracted as a single block of prose. This cue, together with three other formatting features, tell the classes apart at 0.996 AUC. To distinguish benchmarks that contain such a signal from detectors that rely on one, we introduce format counterfactuals: just re-wrapping a machine-written abstract without changing a single word reverses 20-27% of a public detector's correct decisions. Normalizing formatting before training reduces the generalization gap from 0.32 to 0.22. On format-clean benchmarks, it also raises out-of-domain macro-F1 from 0.33 to 0.65 for detectors trained on artifact-containing subsets.
| Comments: | Accepted to DocInsights workshop at EMNLP 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.13583 [cs.CL] |
| (or arXiv:2610.13583v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13583 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Anastasia Voznyuk [view email]
[v1]
Fri, 9 Oct 2026 13:34:45 UTC (59 KB)
来源：arXiv cs.CL · arxiv.org