arXiv cs.AI· Heimo M\"uller, Dominik Steiger, Markus Plass, Andreas Holzinger·· 14 小时前AI 评分29
研究者提出 System Hallucination Scale（SHS），从用户视角评估大语言模型可靠性
The System Hallucination Scale (SHS): A Human-Centered Instrument for Assessing Perceived Reliability of Large Language Models
AI 导读
研究者提出 System Hallucination Scale（SHS），一个仿照 SUS 与 SCS 设计的轻量工具，用 10 个 Likert 量表题从事实准确性、来源可靠性、逻辑连贯性、呈现欺骗性和对用户引导的响应性五个维度衡量用户感知的 LLM 可靠性，并给出方向性得分与一致性诊断。
正文 · 原文
Abstract:Large language models routinely produce factually incorrect or misleading outputs, yet conventional automated benchmarks overlook how users perceive reliability during interaction. We introduce the System Hallucination Scale (SHS), a lightweight instrument modeling the famous System Usability Scale (SUS) and System Causability Scales (SCS). SHS measures perceived reliability across five dimensions: factual accuracy, source reliability, logical coherence, deceptiveness of presentation, and responsiveness to user guidance, using ten balanced Likert-scale items that provide a directional score and consistency diagnostic. In a supervised validation study with 193 participants (386 evaluations across 47 trained experimenters), SHS demonstrated strong internal consistency and high clarity, completing in an average of 4.2 minutes. Participants rated source verifiability and presentation deceptiveness worse than factual accuracy, indicating perceived vulnerabilities lie primarily in traceability and framing. As a proof of concept, SHS assesses human-perceived reliability rather than ground-truth correctness, providing open-source multilingual implementations and an interactive calculator.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.09989 [cs.CL] |
| (or arXiv:2603.09989v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2603.09989 arXiv-issued DOI via DataCite |
|
| Related DOI: | https://doi.org/10.1080/10447318.2026.2739758
DOI(s) linking to related resources |
Submission history
From: Heimo Müller [view email]
[v1]
Fri, 13 Feb 2026 10:13:09 UTC (400 KB)
[v2]
Thu, 29 Oct 2026 13:02:53 UTC (85 KB)
来源：arXiv cs.AI · arxiv.org