arXiv cs.AI· Hangle Hu, Chenyu Hou, Bin Cao, Jing Fan, Wei Luo·· 10 小时前AI 评分31
语言模型在自然语言数据查询中的决策覆盖问题:UDC 与 GCG 诊断协议
Asking but Missing: Decision Coverage in Natural-Language Data Querying
AI 导读
研究提出 Underspecified Decision Completion(UDC)概念,指语言模型在将自然语言问题转为可执行程序时,未识别问题存在的多种计算解释便默默采用假设,导致结果出错。
正文 · 原文
Abstract:Large language models are increasingly used to translate natural language questions about structured data into executable programs. A fundamental challenge is recognizing when a question admits multiple computational interpretations and identifying the decisions needed to resolve them. We investigate whether language models recognize these unresolved decisions before generating code, or silently adopt assumptions that may lead to incorrect results. We refer to the latter behavior as Underspecified Decision Completion (UDC). To distinguish different causes of UDC, we introduce Grounded Code Generation (GCG), a diagnostic protocol that separates the identification of unresolved decisions, their resolution using evidence, and their translation into executable code. By examining each stage separately, we can determine where the decisions responsible for incorrect results were first missed or mishandled. We evaluate this framework on BIRD-Pandas with twelve models. The results show that answering clarification questions consistently improves execution accuracy, yet substantial errors remain. Our findings suggest that the main difficulty lies not in answering clarification questions, but in recognizing which questions need to be asked in the first place.
| Subjects: | Artificial Intelligence (cs.AI); Software Engineering (cs.SE) |
| Cite as: | arXiv:2611.23864 [cs.AI] |
| (or arXiv:2611.23864v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23864 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hangle Hu [view email]
[v1]
Wed, 18 Nov 2026 05:13:13 UTC (632 KB)
来源:arXiv cs.AI · arxiv.org