arXiv cs.CL· Mingyang Song, Mao Zheng, Zhiyuan Ma, Xinpeng Zhou·· 12 小时前AI 评分31
研究揭示 Jev 风格 System One 模型的选择偏差：结构未必带来优势
Understanding Jev-Style System One Models: Does Structure Reduce Selection Bias?
AI 导读
针对两个 Jev 版本、十二个 Jev 复现模型和六个生成式 LLM 的测试显示，Jev 在 1,700 道题上准确率波动不足两个百分点，但打乱选项顺序后累计失败集比原始顺序扩大 27–29%，初始答对的题目在后续呈现中持续失败。这一损失明显大于固定顺序重复，且 Jev 在保持正确选择上并不普遍优于 LLM 基线；投票仅带来有限增益，即使全票一致的答案也经常出错。
正文
Abstract:Selection bias in large language models (LLMs) can persist despite instructions to ignore presentation order or prompts that elicit reasoning. Recent efforts to reproduce Jev-style System One behavior have produced a growing family of models that predict probabilities directly over supplied candidates. Yet systematic comparisons of selection bias across Jev versions, Jev reproductions, and generative LLMs remain limited in multi-option tasks. This leaves a central question: does Jev-style decision structure reduce selection bias? We address this question with a simple test that shuffles options while keeping question and candidate content fixed. Across two Jev versions, twelve Jev reproductions, and six generative LLMs, we trace individual answers through eight presentations and compare them with fixed-order repetition. On our primary collection of 1,700 questions from two benchmarks, Jev's accuracy varies by less than two percentage points, yet its cumulative failure set expands by 27--29\% relative to the original order, with first failures continuing into the final presentation. This loss of initial successes is substantially larger than under fixed-order repetition, even for highly repeatable Jev reproductions. Nor does Jev show a general advantage over the LLM baselines in preserving correct choices. Aggregation offers limited protection: voting yields modest gains for Jev, while even unanimous answers are frequently wrong. These results expose a gap between repeatable decisions and decisions that remain correct across equivalent presentations, showing why stable accuracy alone cannot establish resistance to selection bias.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.35826 [cs.CL] |
| (or arXiv:2610.35826v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35826 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mingyang Song [view email]
[v1]
Mon, 26 Oct 2026 17:48:55 UTC (1,929 KB)
来源：arXiv cs.CL · arxiv.org