arXiv cs.CL· Stanislav Kirichenko·· 11 小时前AI 评分31
同一来源、不同判定：文本生产条件干扰俄语机器生成文本检测
Same Origin, Different Verdicts: Text Production Conditions Confound Machine-Generated Text Detection in Russian
AI 导读
一项针对 1882 篇俄语文档的研究发现，文本生产条件本身会干扰机器生成文本检测：在机器语料内部固定来源、仅改变任务措辞，四种机器相似度度量中有三种得分负向偏移、judge 模型却正向偏移，且四个置信区间均不含零。
正文 · 原文
Abstract:Automated detection of machine-generated text is applied where the verdict carries a cost for the author, yet origin is a property of the production process, not an observable property of the text. The conditions under which a text is produced push it in the same direction in which a generative model is presumed to push it. This study measures the contribution of these conditions while origin is held fixed. The Russian-language corpus of 1882 documents comprises a machine part, generated by four generation channels (each backed by its own model family) across 45 tasks in three task modes, and an archival human part annotated for source regimentation level before the texts were read. The primary contrast is computed within the machine corpus: both halves of a pair are generated by the same model for the same task and differ only in task wording. Machine-likeness is measured by four operationalisations: an interpretable index with weights fixed before data collection, a trained classifier, a zero-shot metric on an open model, a judge model. The main result is disagreement between procedures: changing the task mode shifted the scores of all four, but three gave a negative shift, the judge a positive one, and none of the four intervals covers zero. The comparison plan and the specifications of procedures 2-4 were fixed after the result of procedure 1, before procedures 2-4 ran; the disagreement survived correction of a corpus defect and of the nested validation. A baseline on the source identifier alone reaches a median AUROC of 1.0000 on the thirteen group splits where it is defined, and removing emphasis markers flipped a confident machine verdict to human without changing the full text by a single character. The scores are properties of a specific measurement system and corpus: they reflect data collection, repetition, formatting, and preprocessing decisions as well as linguistic style.
| Comments: | 65 pages, 6 figures, 24 tables. English translation of the frozen Russian manuscript; the Russian text is the source of record and is available with all data, code and run manifests at this https URL (v1.0.1) |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.05385 [cs.CL] |
| (or arXiv:2611.05385v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05385 arXiv-issued DOI via DataCite |
Submission history
From: Stanislav Kirichenko [view email]
[v1]
Mon, 3 Aug 2026 09:03:28 UTC (115 KB)
来源：arXiv cs.CL · arxiv.org