arXiv cs.CL· Xing Li, Qingcheng Chang, Jinzhong Ning, Changfeng Xu, Shenlong Zhang, Yijia Zhang, Ling Luo, Hongfei Lin·· 9 天前AI 评分27
专用决策模型 Jev 与通用 LLM 对比：知识、推理与多语言任务基准测试
Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks
AI 导读
专用"System One"决策模型 Jev 在 13 个多项选择基准上与 19 个 LLM 对比，在知识与常识任务上媲美前沿 LLM，并在 MMLU-Redux 和 ARC-Challenge 上取得最佳成绩。
正文 · 原文
Abstract:Jev is a "System One" model that returns a choice among given options instead of generating text. We study how such a specialized decision model compares with general-purpose large language models (LLMs). We evaluate Jev on 13 multiple-choice benchmarks covering knowledge, reasoning, and multilingual understanding, and compare it with 19 LLMs in three tiers: frontier, representative, and small. Jev is competitive with frontier LLMs on knowledge and commonsense benchmarks and obtains the best score on MMLU-Redux and ARC-Challenge. Outside mathematics, it also outperforms most representative LLMs and all small LLMs. However, it falls behind on mathematical word problems: on MathQA, it is 17.7 points below the frontier median and scores lower than all 19 LLMs. These results indicate that a specialized decision model can match general-purpose LLMs on decisions that rely mainly on knowledge, but not on decisions that require multi-step calculation.
| Comments: | 10 pages, 2 figures, 7 tables |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.11978 [cs.CL] |
| (or arXiv:2610.11978v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11978 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xing Li [view email]
[v1]
Thu, 8 Oct 2026 13:52:42 UTC (89 KB)
来源：arXiv cs.CL · arxiv.org