arXiv cs.AI· Yuhan Tang, Ke Fan, Xinkai Zou, Baichuan Mo, Kangxin Cui, Ao Qu, Ou Zheng, Jiangbo Yu, Haris N. Koutsopoulos, Jinhua Zhao, Yuan Zhou·· 10 小时前AI 评分42
Ask2Decide 基准：评估 LLM 智能体能否通过提问获取决策相关信息
From Questions to Decisions: Benchmarking How LLM Agents Acquire Decision-Relevant Information from Human Beings
AI 导读
研究者推出 Ask2Decide 基准，含 43 个规划任务，用优化求解器判定哪些缺失事实会改变最优方案，并加入不改变方案的相似表述事实作干扰项。基于 19 名参与者的 79 次交互，校准出更贴近人类回复模式的模拟操作员。四个前沿 LLM 在信息缺失时初始方案不可行率达 58-98%，且找出关键事实的比例（39-52%）仅略高于无关事实（32-41%）。
正文 · 原文
Authors:Yuhan Tang, Ke Fan, Xinkai Zou, Baichuan Mo, Kangxin Cui, Ao Qu, Ou Zheng, Jiangbo Yu, Haris N. Koutsopoulos, Jinhua Zhao, Yuan Zhou
Abstract:LLM agents are increasingly asked to make plans for people, such as scheduling drivers or assigning nurses. However, people's requests are often incomplete, and the supporting information for decision-making may be outdated, requiring agents to ask for clarification. Existing benchmarks typically provide fully specified problems or rely on uncalibrated LLM-simulated users, leaving this ability insufficiently tested. We introduce Ask2Decide, a benchmark of 43 planning tasks that evaluates whether agents ask for missing information that matters for their decisions. An operator holds facts absent from the agent's request and records, and optimization solvers determine which facts change the optimal plan. We also include similarly worded facts that leave the optimal plan unchanged to test whether agents distinguish useful information from irrelevant details. We collect 79 interaction sessions from 19 participants and find that, after an incorrect plan, human replies omit the correcting fact far more often than replies from an uncalibrated simulated user (43% versus 7%). Using these interactions, we calibrate a simulated operator that more closely matches human reply patterns. Evaluating four frontier LLMs, we find that their initial plans are infeasible on 58-98% of tasks when information is missing, compared with 5-28% when all facts are provided upfront. Agents uncover facts that change the optimal plan only slightly more often than facts that leave it unchanged (39-52% versus 32-41%). These results show that obtaining decision-relevant information through interaction remains a major bottleneck for LLM planning. This is ongoing work and will be updated. The date of this version is shown on the first page.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.13015 [cs.AI] |
| (or arXiv:2611.13015v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13015 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuhan Tang [view email]
[v1]
Mon, 9 Nov 2026 04:22:15 UTC (66 KB)
来源：arXiv cs.AI · arxiv.org