arXiv cs.CL· Jieyu Zhou, Zijian Wang, Yun Wang·· 12 小时前AI 评分38
IntentDelta：面向演化用户意图的人类构建基准
IntentDelta: A Human-Constructed Benchmark for Evolving User Intent
AI 导读
研究者发布 IntentDelta，一个覆盖 WebShop 与 TravelPlanner、含 638 条轨迹和 3,766 轮对话的人类构建多轮基准，每轮配对用户话语、累积意图状态与人类参考动作。
正文
Abstract:Language-model agents must adapt as users revise their requirements and priorities during complex tasks. We introduce IntentDelta, a human-constructed multi-turn benchmark with 638 trajectories and 3,766 turns across WebShop and TravelPlanner. Each turn pairs a user utterance with a cumulative intention state and human reference actions. The trajectories are cross-verified and cover feasibility- and preference-driven changes in constraints and priorities. We evaluate eleven agents on intention recovery and action quality. Average Intention Recovery is 33.1\% in TravelPlanner and 44.7\% in WebShop, while average Action Score is 31.6% and 41.0%, respectively. Agents struggle to preserve earlier requirements and distinguish soft preferences from must-have constraints. Correct intention recovery also does not guarantee successful execution, highlighting challenges in translating evolving user intentions into actions.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.33806 [cs.CL] |
| (or arXiv:2610.33806v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33806 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jieyu Zhou [view email]
[v1]
Sat, 24 Oct 2026 17:10:48 UTC (1,404 KB)
来源：arXiv cs.CL · arxiv.org