arXiv cs.CL· Shuo Lin, Jinzhe Li, Chenlu Guo, Yanhan Li, Yuan Wu, Yi Chang·· 2 天前AI 评分36
PremiseSQL：评估 Text-to-SQL 系统中的前提批判与拒答能力
PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems
AI 导读
研究者提出 PremiseSQL 基准，包含 1800 条存在前提缺陷的 NL2SQL 查询及对应的 1800 条原始查询，覆盖四类前提失败，用于评估模型识别缺陷查询并拒绝生成误导性 SQL 的能力。实验显示，前沿 LLM 默认高度信任前提、很少批判缺陷前提；强化批判导向的提示词虽能提升检测与拒答，但也会增加误判无效并抑制对原始查询的 SQL 生成。
正文 · 原文
Abstract:Recent advances in large language models (LLMs) have substantially improved Text-to-SQL performance, yet existing NL2SQL systems largely assume that user queries are valid and answerable. As a result, models often generate executable SQL even for queries with flawed premises. We introduce premise critique ability in NL2SQL, namely the ability to recognize flawed queries and abstain from generating misleading SQL. To evaluate this capability, we propose PremiseSQL, a benchmark containing 1,800 flawed NL2SQL queries across four categories of premise failures, along with their 1,800 original queries. We further design an evaluation framework to measure intrinsic and prompt-induced premise-aware behavior across three prompting settings. Experiments on frontier LLMs show that current models are strongly premise-trusting by default and rarely critique flawed premises. Stronger critique-oriented prompting improves detection and abstention on flawed queries, but also increases false invalid judgments and suppresses SQL generation on originals. The code is available at this https URL.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.22158 [cs.CL] |
| (or arXiv:2610.22158v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22158 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shuo Lin [view email]
[v1]
Thu, 15 Oct 2026 16:23:17 UTC (3,196 KB)
来源：arXiv cs.CL · arxiv.org