arXiv cs.CL· Mohammed Nayeem Khan, Logesh L·· 11 小时前AI 评分31
Laya a1 研究：非自回归决策模型的选项顺序敏感性
Maya a1: Do the Options Read the Model, or Does Their Position? Order Sensitivity in a Non-Autoregressive Decision Model
AI 导读
研究者重新工程化 Laya 检查点（421M 参数、2 层选项头、无位置张量），测试选项排列是否影响答案。在 951 道 Qwen 验证题上循环轮换选项后，25.8% 的问题答案发生翻转，金标选项置首与置尾的准确率相差 10.1 分，翻转集中在模型低置信度区间（54% 对 8%）。
正文 · 原文
Abstract:Laya answers typed multiple-choice questions in one forward pass by scoring a hidden state at one [MASK] marker per option. We re-engineer the checkpoint (421M parameters; a 2-layer option head with no positional tensor; a documented but unused parallel layout) and test whether the answer depends on where an option is listed. Presenting 951 Qwen-verified questions under every cyclic rotation of their options (or six random permutations for 12 options), the released checkpoint changes its answer for 25.8% [22.9-28.7] of questions (19.7% on the 605 English-language items alone); accuracy with the gold option first versus last differs by 10.1 points, and the flips concentrate where the model's own confidence is low (54% versus 8% across confidence terciles). Fine-tuning with Laya's trainer on 2,818 teacher-verified rows with option shuffling cuts the flip rate to 7.2%; the identical rows without shuffling give 11.4%, so shuffling is responsible for 4.2 points of the improvement. A confidence-weighted vote over orderings gives 69.7% against 69.1% for a single base pass at 4.1x the compute. Labels are LLM-made and the test sets are teacher-filtered. We also fine-tune with the authors' parallel layout on the same rows: flip rate 0.5%.
| Comments: | 24 pages, 13 figures, 23 tables |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.09803 [cs.AI] |
| (or arXiv:2611.09803v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.09803 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mohammed Nayeem Khan [view email]
[v1]
Fri, 6 Nov 2026 17:54:01 UTC (1,616 KB)
来源：arXiv cs.CL · arxiv.org