arXiv cs.CY· Alexander Loth, Martin Kappes, Marc-Oliver Pahl·· 11 小时前AI 评分49
JudgeGPT 研究：人类无法可靠分辨 LLM 生成的新闻
Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News
AI 导读
基于 JudgeGPT 平台收集的 1054 名参与者对六个 LLM 生成内容的 2318 次判断，研究发现人类无法可靠区分机器生成与人类撰写的文本（p > .05），且该结论适用于包括 7B 参数开源模型在内的所有测试模型。自评领域专长可预测判断准确率（r = .35），而政治倾向无显著影响；连续约 30 次评估后准确率因认知疲劳下降。研究认为用户侧检测不可行，需转向加密内容溯源等系统级手段。
正文 · 原文
Abstract:Can humans tell whether a news article was written by a person or a large language model (LLM)? We investigate this question using JudgeGPT, a study platform that independently measures source attribution (human vs. machine) and authenticity judgment (legitimate vs. fake) on continuous scales. From 2,318 judgments collected from 1,054 participants across content generated by six LLMs, we report five findings: (1) participants cannot reliably distinguish machine-generated from human-written text (p > .05, Welch's t-test); (2) this inability holds across all tested models, including open-weight models with as few as 7B parameters; (3) self-reported domain expertise predicts judgment accuracy (r = .35, p < .001) whereas political orientation does not (r = -.10, n.s.); (4) clustering reveals distinct response strategies ("Skeptics" vs. "Believers"); and (5) accuracy degrades after approximately 30 sequential evaluations due to cognitive fatigue. The answer, in short, is no: humans cannot reliably tell. These results indicate that user-side detection is not a viable defense and motivate system-level countermeasures such as cryptographic content provenance.
| Comments: | 6 pages, 6 figures, 1 table. Author version of the paper published in WebSci Companion '26, pp. 34-39 |
| Subjects: | Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Human-Computer Interaction (cs.HC) |
| ACM classes: | H.5.2; H.1.2; K.4.2; I.2.7 |
| Cite as: | arXiv:2604.03755 [cs.CY] |
| (or arXiv:2604.03755v2 [cs.CY] for this version) | |
| https://doi.org/10.48550/arXiv.2604.03755 arXiv-issued DOI via DataCite |
|
| Journal reference: | Companion Publication of the 2026 18th ACM Web Science Conference (WebSci Companion '26), pp. 34-39 |
| Related DOI: | https://doi.org/10.1145/3795513.3807431
DOI(s) linking to related resources |
Submission history
From: Alexander Loth [view email]
[v1]
Sat, 4 Apr 2026 15:03:42 UTC (1,197 KB)
[v2]
Thu, 29 Oct 2026 22:00:44 UTC (1,178 KB)
来源：arXiv cs.CY · arxiv.org