arXiv cs.CL· Ziyuan Wang, Genliang Zhu, Xiaobo Chen·· 11 小时前AI 评分52
研究审计 AI 文本改写工具：事实与检测表现如何
Humanized but Still True? Auditing Facts and Detection in AI-Text Humanizers
AI 导读
一项 arXiv 论文审计 AI 文本改写工具（humanizer）是否保留可核查细节。在 2026 年 10 月的公开榜单周期中，11 款商业工具全部在 35-65% 含数字、日期、人名或引文的文档里改动或删除了至少一项，多数改动落在引文上；仅统计数字和人名时，11 款中仍有 10 款在 20% 以上文档中改动至少一项。
正文 · 原文
Abstract:AI "humanizers" rewrite machine-generated text so that detectors classify it as human-written. Public leaderboards score evasion and embedding similarity, but not whether checkable details survive. We check whether numbers, dates, person and organization names, and quoted spans survive each rewrite, using a rule-based checker validated by two AI annotators (precision 0.84-1.00, recall 0.93-0.95; no human annotation). In the October 2026 cycle of a public humanizer leaderboard, all 11 commercial tools changed or dropped at least one of these units in 35-65% of the documents that contained one. Most changes were to quoted spans; counting numbers and names alone, 10 of the 11 tools still altered at least one in 20% or more of the documents. The September 2026 cycle was similar (19-50%). An open 12B humanizer kept at least 97% of numbers and names in fact-dense U.S. federal agency text, but reworded 67-78% of the quoted spans, against its own prompt. Unit retention and detector evasion were not clearly related across tools (Spearman rho = -0.46, 95% CI -0.65 to 0.08), and the open model's four released files (two of them quantization-aware low-bit files) did not show the pre-registered monotone trade-off. On federal texts with matched human, AI-styled and humanized versions, a general-purpose decision model, Jev, asked a single zero-shot question, separated human originals from humanized rewrites with AUROC 0.864 (0.827-0.902), 0.18-0.22 higher than Binoculars, Fast-DetectGPT and the adversarially trained RADAR, while calling 24% of the human originals AI at a 0.5 threshold. On the AI-styled rewrites by 2026 models, both perplexity-based detectors scored below chance (AUROC 0.37 and 0.39), also with line breaks removed. Hypotheses, data rules and analyses were frozen before the main runs; two earlier data contacts are disclosed. We release the checker and the per-item results.
| Comments: | 11 pages, 2 figures, 4 tables. Code and data: this https URL |
| Subjects: | Computation and Language (cs.CL); Cryptography and Security (cs.CR) |
| ACM classes: | I.2.7 |
| Cite as: | arXiv:2611.00359 [cs.CL] |
| (or arXiv:2611.00359v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.00359 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ziyuan Wang [view email]
[v1]
Fri, 30 Oct 2026 20:36:08 UTC (57 KB)
来源：arXiv cs.CL · arxiv.org