arXiv cs.AI· Qiuyi (Richard), Zhang, Zhen Xu, Jing Gu, Seojin Bang, Arnav Mohanty Das, Yaning Hu, Gautam Kumar, Jared Lichtarge, Le Xue, Bettina Catoto, Akul Datta, Daniel Luo, Farha Khan, Marcella Valentine, Yinfei Yang, Andrew M. Dai·· 14 小时前AI 评分36
DecoyCount：评估计数任务中的视觉-推理错位
DecoyCount: Evaluating Visual-Reasoning Misalignment in Counting
AI 导读
研究团队推出 DecoyCount 基准，基于 1,660 张真实图像和七种诱饵机制，配套中性问题、对齐提示与多跳多图问题三种提示，用于区分计数错误源于感知还是推理。前沿模型在该基准上准确率仍低于 50%，推理-对齐差距达 11.8-22.1 个百分点，且能力更强的模型、更大的思考预算与智能体流程往往带来更大差距。研究结论是更强的通用推理并不能自动泛化到对齐的视觉推理。
正文
Authors:Qiuyi (Richard)Zhang, Zhen Xu, Jing Gu, Seojin Bang, Arnav Mohanty Das, Yaning Hu, Gautam Kumar, Jared Lichtarge, Le Xue, Bettina Catoto, Akul Datta, Daniel Luo, Farha Khan, Marcella Valentine, Yinfei Yang, Andrew M. Dai
Abstract:Visual counting benchmarks report whether vision-language models produce the correct number, but not whether errors arise from perception or reasoning. We introduce DecoyCount, a benchmark built from 1,660 real-world images with designed decoys across seven mechanisms, along with paired prompts: a neutral question, a matched alignment cue, and a novel multi-hop, multi-image question that situates the same counting problem in a realistic scenario. Using this design, we find that counting failures are induced partly by visual misalignment: models can execute the intended counting process when cue-guided but fail to select it unaided, and we term this difference in accuracy the reasoning-alignment gap. Despite being highly capable on general reasoning, frontier models remain below 50\% accuracy on our benchmark, while alignment gaps range from 11.8-22.1 percentage points; strikingly, we observe that more capable models, greater thinking budgets, and agentic processes often induce larger gaps. Because humans may bring context that models lack, we test whether textual underspecification contributes to misalignment using a scenario-based multi-hop condition without explicit hinting, but this added context does not improve aggregate performance. Our analysis concludes that stronger general reasoning does not by itself generalize to aligned visual reasoning, leaving misalignment with human visual expectations as a substantial component of the remaining deficit.
| Subjects: | Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2610.34855 [cs.AI] |
| (or arXiv:2610.34855v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34855 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Qiuyi (Richard) Zhang [view email]
[v1]
Mon, 26 Oct 2026 05:19:02 UTC (2,181 KB)
来源：arXiv cs.AI · arxiv.org