arXiv cs.CL· Rongyu Zhang, Mutian Bao, Jinjian Zhang, Yuetian Du, He Xu, Wei Zhou, Xiaojun Zhu, Ming Kong, Jie Liu, Linjian Mo, Qiang Zhu·· 11 小时前AI 评分38
VagueBench 与 DialogGRPO：评测并训练开放域 LLM 对话中的主动追问能力
Don't Guess, Ask Wisely: Benchmarking and Training Proactive Elicitation in Open-Domain LLM Conversations
AI 导读
研究团队提出 VagueBench 基准与 DialogGRPO 离线强化学习方法，用于评测和训练 LLM 在用户查询不完整时的主动追问能力。
正文 · 原文
Authors:Rongyu Zhang, Mutian Bao, Jinjian Zhang, Yuetian Du, He Xu, Wei Zhou, Xiaojun Zhu, Ming Kong, Jie Liu, Linjian Mo, Qiang Zhu
Abstract:When user queries are incomplete, large language models should ask targeted questions before committing to an answer, a capability we call proactive elicitation, yet evaluation on real user-assistant queries across seven LLMs (8B to frontier) shows that fewer than 11% of underspecified queries trigger a follow-up question. To benchmark and train this capability, we introduce VagueBench, where each seed query is annotated with structured intent and detail units then degraded into intent-vague and fully-vague variants via controlled removal of linguistic cues, yielding known hidden information sets for evaluating both when and what to ask. We propose DialogGRPO, an offline reinforcement learning method that organizes multi-turn elicitation as a dialogue-state directed acyclic graph, where candidate questions at each node receive dense turn-level rewards combining immediate information gain with lookahead value through group-relative comparison, enabling fine-grained credit assignment without runtime simulation. DialogGRPO elicits missing information on over 88% of vague queries with substantial gains in both recovery efficiency and answer coverage, without over-asking on complete queries, outperforms prompting and test-time-scaling baselines, and generalizes zero-shot to MultiWOZ 2.4 and AmbigQA.
| Comments: | 26 pages, 6 figures; EMNLP 2026 oral |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.01878 [cs.CL] |
| (or arXiv:2611.01878v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.01878 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Rongyu Zhang [view email]
[v1]
Sun, 1 Nov 2026 13:15:14 UTC (2,124 KB)
来源：arXiv cs.CL · arxiv.org