arXiv cs.CL· Boyuan Deng, Shuyi Fan, Hongyang Zhang, Xinhong Xie·· 2 天前AI 评分22
Jev 用于科学决策：评估语义选择及其后果
Jev for Scientific Decisions: Evaluating Semantic Choices and Their Consequences
AI 导读
研究将 Jev 作为语义决策组件进行评测，在 10 个科学案例的 20 个 Choices 上比较 12 种模型配置，每个重复 5 次，分别测量语义选择、下游输出与最终结论标签。Jev 在语义完全正确率上与另外 5 种配置持平，并在成功响应中取得最低中位延迟。三个对比模型在一个 culture-history 问题上出现 7 次错误选择，改变了下游计数却保留了正确的最终标签。
正文 · 原文
Abstract:Scientific workflows often require choosing among known relations before a deterministic calculation can proceed. Whether observations share a culture, treatment or reference standard can change the scientific meaning of the resulting count or comparison. We evaluate Jev as a semantic decision component using a harness that follows its documented guidance and assigns arithmetic to code. The study compares twelve model configurations on twenty source-grounded Choices across ten scientific cases, each repeated five times. We measure semantic selections, downstream outputs and final claim labels separately. Jev matched five other configurations at complete semantic correctness and achieved the lowest observed median latency among successful responses. Across three comparison models, seven wrong selections on one culture-history question changed downstream counts while preserving the correct final label. These results identify a useful role for Jev in prepared scientific decision tasks and show why evaluating that role requires checking the relations and quantities that a workflow will reuse.
| Comments: | 9 pages, 1 figure, 6 tables. Code available at this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| ACM classes: | I.2.7 |
| Cite as: | arXiv:2609.24965 [cs.CL] |
| (or arXiv:2609.24965v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.24965 arXiv-issued DOI via DataCite |
Submission history
From: Boyuan Deng [view email]
[v1]
Mon, 21 Sep 2026 17:51:50 UTC (167 KB)
[v2]
Wed, 23 Sep 2026 04:04:31 UTC (166 KB)
[v3]
Thu, 15 Oct 2026 07:14:47 UTC (218 KB)
来源：arXiv cs.CL · arxiv.org