arXiv cs.AI· Tianyou Lai, Wentao Yue, Zhiying Deng, Qilei Li·· 3 天前AI 评分28
Choice2Seq：衡量模型在序列构建中正确性保留的基准
Correct Choices, Failed Sequences: Choice2Seq Benchmarks Correctness Retention Across Output Interfaces
AI 导读
研究者提出诊断基准 Choice2Seq，用序列组合率（SCR）衡量模型在完整答案选择与逐步序列构建之间的正确性保留。Jev 在算术和词汇检索上选择准确率达 97.1% 和 99.5%，但在序列构建下仅保留 33.0% 和 2.1% 的正确选择，StartLux 和 Qwen 也存在明显差距。将局部选项限制为候选支持的字符可改善构建，显式位置索引可提升复制保真度。
正文 · 原文
Abstract:A model may select the correct answer yet fail to construct it through a sequence of local decisions. We introduce Choice2Seq, a diagnostic benchmark that measures correctness retention across complete-answer selection and sequential construction in independent runs on the same problems and candidate sets. Sequence Composition Rate (SCR) measures the fraction of correctly selected items that are also answered correctly through local construction. Jev achieves 97.1% and 99.5% selection accuracy on arithmetic and lexical retrieval, yet retains only 33.0% and 2.1% of those correct choices under serial construction, respectively. Substantial gaps also emerge for StartLux and Qwen, including when Qwen emits characters directly. Restricting local options to candidate-supported characters improves construction, while explicit position indexing improves copying fidelity, even for incorrect selections. These interventions distinguish task correctness, plan fidelity, and output compliance. Matched dialogue-state evaluations reveal that local field construction largely preserves Jev's correct states but reduces Qwen's mean-turn accuracy, highlighting task- and interface-dependent differences. Sequence-level confidence and request-cost analyses further characterize completed outputs.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.16191 [cs.AI] |
| (or arXiv:2610.16191v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.16191 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tianyou Lai [view email]
[v1]
Mon, 12 Oct 2026 05:11:37 UTC (947 KB)
来源：arXiv cs.AI · arxiv.org