arXiv cs.CL· Zongwan Cao, Bingbing Wen, Lucy Lu Wang·· 12 小时前AI 评分29
Clarify-or-Answer：面向上下文信息不足的智能体 VQA 强化学习方法
Clarify or Answer: Reinforcement Learning for Agentic VQA with Context Under-specification
AI 导读
研究者提出 Clarify-or-Answer（CoA）智能体，将"是否提问"与"问什么"分开建模，在图像信息不足时生成澄清问题迭代补全上下文。配套发布 CONTEXTCLARIFY 歧义 VQA 数据集及 GRPO-CR 强化学习方法，用多路奖励信号优化澄清问题质量。在三个 VLM 和三个数据集上，CoA 在模块与系统层面均取得一致提升，单轮澄清即可带来显著收益。
正文 · 原文
Abstract:Real-world visual question answering (VQA) may be context-dependent: an image-question pair could be under-specified, such that the correct answer depends on external information that is not observable in the image. In such cases, directly answering can lead to confident but incorrect predictions. We propose Clarify-or-Answer (CoA), an ask-or-answer agent that separately models the decision to ask or answer, and what to ask if needed. CoA first determines whether clarification is necessary; if so, it generates one or more clarification questions to iteratively enrich the context until ambiguity is resolved. To validate this framework, we introduce CONTEXTCLARIFY, a dataset of ambiguous VQA questions along with a contrast set of non-ambiguous questions. We further introduce GRPO-CR(Clarification Reasoning), a reinforcement learning approach that optimizes clarification question generation with multiple reward signals encouraging well-formed, focused, non-trivial questions that resolve ambiguity. Across three VLMs and three datasets, CoA achieves consistent improvements at both the module and system levels, with a single clarification turn providing substantial gains and recursive multi-turn execution offering modest additional improvements in select settings.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2601.16400 [cs.CL] |
| (or arXiv:2601.16400v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2601.16400 arXiv-issued DOI via DataCite |
Submission history
From: Zongwan Cao [view email]
[v1]
Fri, 23 Jan 2026 02:12:33 UTC (3,243 KB)
[v2]
Tue, 20 Oct 2026 03:22:00 UTC (3,206 KB)
来源：arXiv cs.CL · arxiv.org