arXiv cs.AI· K M Sajjadul Islam, Pranav Yaramalla, Nadiyah Johnson, Mark Flower, Praveen Madiraju, Zeno Franco·· 10 小时前AI 评分29
退伍军人心理健康危机严重度标注：LLM 比人类专家更一致吗？
Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts?
AI 导读
一项研究用 149 条推文和五级严重度分类，对比 15 个开源与 10 个专有 LLM 在八种提示词变体下与 8 位退伍军人领域专家的标注表现。LLM 间序数一致性 α 在基线提示词下达 0.8336，高于人类专家的 0.494，GPT-5.6-Terra 整体表现最佳，开源 Qwen2.5-7B 在识别急性危机伴自杀意念案例上最强。
正文 · 原文
Abstract:Large language models (LLMs) offer a scalable approach for annotating mental health and crisis-related social media content, but their reliability in safety-critical tasks remains unclear. This study evaluates LLM-based crisis-severity annotation of veteran-related social media posts against annotations from eight Veteran Subject Matter Experts (SMEs). Using 149 tweets and five severity categories, we evaluate 15 open-source and 10 proprietary LLMs across eight prompt variants, examining annotation performance, inter-model agreement, prompt sensitivity, and high-severity crisis detection. Human reviewers showed higher ordinal than nominal agreement ({\alpha} = 0.494), indicating that disagreements often occurred between neighboring severity levels. In contrast, selected LLMs demonstrated substantially higher inter-model consistency, with ordinal agreement ({\alpha}) reaching 0.8336 under the baseline prompt. Proprietary models generally showed better agreement with expert annotations and greater stability than open-source models. GPT-5.6-Terra achieved the best overall performance, while the open-source Qwen2.5-7B showed the strongest performance in identifying Acute Crisis with Suicidal Ideation cases. However, higher model consistency did not necessarily correspond to agreement with human judgments or reliable detection of rare high-severity cases. Label-flipped demonstration experiments also revealed model-specific sensitivity to misleading in-context examples, while explanatory demonstrations did not consistently improve performance. Overall, LLMs show promise as scalable complements to expert annotation, but require careful evaluation before use in high-stakes mental health applications.
| Comments: | Accepted in IEEE Big Data 2026 |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.10048 [cs.AI] |
| (or arXiv:2611.10048v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.10048 arXiv-issued DOI via DataCite |
Submission history
From: K M Sajjadul Islam [view email]
[v1]
Thu, 5 Nov 2026 22:18:51 UTC (209 KB)
[v2]
Tue, 10 Nov 2026 03:19:07 UTC (209 KB)
来源：arXiv cs.AI · arxiv.org