arXiv cs.CL· Xiaoshan Zhou, Zaifu Zhan·· 10 天前AI 评分31
CrisisFake：AI 生成灾害社交媒体文本检测的基准有效性研究
CrisisFake: Benchmark Validity of AI-Generated Text Detection for Disaster Social Sensing
AI 导读
研究构建了 CrisisFake 数据集，包含基于 Qwen2.5-7B 生成的 12,000 条文本，覆盖九场灾害的 3,000 个语义单元，用于检验文本检测器能否区分人类与 LLM 生成的灾害帖子。
正文 · 原文
Abstract:Disaster social sensing converts public social-media posts into evidence for situational awareness and humanitarian response, but plausible LLM-generated posts can contaminate this information stream and distort assessments of needs, damage, and resource priorities. This study empirically investigates whether text-based detectors can distinguish human-authored from LLM-generated disaster posts and what textual cues underlie their judgments. We construct CrisisFake, a Qwen2.5-7B-based dataset of 12,000 texts organized into 3,000 matched semantic units from nine disasters. Each unit contains an original human post, a minimally LLM-proofread human post, a fact-preserving LLM-generated post produced using LoRA, and an affectively reframed version of the artificial post. A separate 6,000-text corpus spanning 42 disaster events is independently constructed to support model selection and threshold calibration. We evaluate OSM-Det, Fast-DetectGPT, Binoculars, and direct LLM judges across five model families, and further examine how disaster-domain calibration and superficial linguistic cues, such as retweet markers, user mentions, hashtags, URLs, punctuation, and text length, affect detector performance. Across fourteen frozen configurations, AUROC ranges from 0.402 to 0.517, while the best prospective recall at a calibration-derived low-false-positive operating point is only 3.6%, indicating near-chance discrimination. For OSM-Det, a disaster-calibrated linear head improves AUROC to 0.817; however, a seven-feature textual classifier alone reaches AUROC 0.784 on the original-versus-factual-LLM contrast, and neutralizing identified surface asymmetries reduces the corresponding linear-head AUROC from 0.733 to 0.594. These findings provide empirical evidence that LLM-generated text detection is largely driven by linguistic cues and remains insufficiently robust for short-form disaster social media.
| Subjects: | Computation and Language (cs.CL); Computers and Society (cs.CY) |
| Cite as: | arXiv:2609.35821 [cs.CL] |
| (or arXiv:2609.35821v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.35821 arXiv-issued DOI via DataCite |
Submission history
From: Xiaoshan Zhou [view email]
[v1]
Mon, 21 Sep 2026 21:20:27 UTC (829 KB)
[v2]
Wed, 7 Oct 2026 14:47:24 UTC (1,056 KB)
来源：arXiv cs.CL · arxiv.org