arXiv cs.CL· Sneha Oram, Tejomay Kishor Padole, Pushpak Bhattacharyya, Suyash P. Awate·· 13 小时前AI 评分31
PRiME：心理健康领域的语用推理研究
PRiME: Pragmatic Reasoning In MEntal health
AI 导读
研究构建三个数据集与三项任务，评估12个LLM在心理健康高风险场景下的语用推理能力，Mistral-7B表现最佳，整体与人类评估相当。团队还用误差不对称比率分析模型表现，并对MentaLLaMA-7B做了一阶注意力 rollout 实验，公开模型的架构行为结果。
正文 · 原文
Abstract:In this work, the task and domain interaction of reasoning and mental health are investigated. As large language models (LLMs) present exceptional text understanding and fluent text generation capability, we analyze whether they can reason in the high-stakes domain of mental health. We begin by introducing three datasets and three tasks leveraging the existing work in pragmatic reasoning. The analysis constitutes an evaluation of $12$ LLMs, including MentaLLaMA, which is a mental health reasoning and explanation generation model. The performance of the models remains on par with human evaluation, with the Mistral-7B model giving the best performance. We further analyze the performance of the models with error asymmetry ratio and conduct first-order attention rollout experiment with MentaLLaMA-7B. We present the results for the community reporting the architectural behaviour of the models for the input text and recommend further careful analysis for the high-stakes domain.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2507.23247 [cs.CL] |
| (or arXiv:2507.23247v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2507.23247 arXiv-issued DOI via DataCite |
Submission history
From: Sneha Oram [view email]
[v1]
Thu, 31 Jul 2025 05:10:38 UTC (7,303 KB)
[v2]
Fri, 7 Nov 2025 17:49:33 UTC (458 KB)
[v3]
Wed, 21 Oct 2026 01:48:06 UTC (12,717 KB)
来源：arXiv cs.CL · arxiv.org