arXiv cs.CL· Yanrui Du, Danyang Zhao, Sendong Zhao, Wenhao Li, Yibo Gao, Haochun Wang, Zhuoer Xu, Shiwen Cui, Bing Qin·· 12 小时前AI 评分34
Jev 类系统一决策模型在医学场景有多可靠？与 LLM 的对比研究
How Reliable Are System-One Decisions in Medicine? A Comparative Study of Jev, Jev-like Models, and LLMs
AI 导读
一项对比研究在 15 个医学数据集、8,742 个实例上评估了 Jev、Jev-like 模型与 LLM 的系统一决策可靠性。Jev 在文本驱动任务上可与领先模型匹敌，但在数值驱动任务上表现较弱，暴露出能力边界；面对问题格式变化，Jev 决策一致性较强、准确率波动小，StartLux-27B 一致性相当，D3-27B 总体准确率更高且平均决策反转率更低。
正文 · 原文
Abstract:Jev has attracted considerable attention for its rapid, generalizable decision-making, yet its medical reliability remains underexplored. To address this gap, our study compares various types of system-one decision models across 15 medical datasets comprising 8,742 instances under a unified protocol. Beyond accuracy, we assess decision stability and the ability to recognize unreliable decisions. Our results show that Jev rivals leading models on primarily text-driven tasks but is less effective on numerically driven tasks, revealing a distinct capability boundary. Under question-format variations, Jev maintains strong decision consistency and small accuracy fluctuations. StartLux-27B exhibits comparable consistency, while D3-27B achieves higher overall accuracy and a lower mean decision reversal rate. But these strengths do not extend to reliable abstention awareness: most evaluated models show limitations in both recognizing missing correct answers and using optional abstention. Moreover, we find that Jev's probability-based answer selection can be improved by changing how decisions are scored: normalizing option-wise binary scores yields higher retained accuracy than choice scoring. Taken together, our findings provide early guidance for improving and applying decision models in medicine.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.18529 [cs.CL] |
| (or arXiv:2610.18529v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.18529 arXiv-issued DOI via DataCite |
Submission history
From: Danyang Zhao [view email]
[v1]
Tue, 13 Oct 2026 15:15:46 UTC (316 KB)
[v2]
Fri, 16 Oct 2026 16:01:04 UTC (327 KB)
来源：arXiv cs.CL · arxiv.org