arXiv cs.CL· Ruoyu Zhang, Zhaocen Liu, Weiyi Tian, Melissa Feng, Shilong Liu·· 12 小时前AI 评分38
OptionWorld：诊断 LLM 长期个人助理能力缺陷的基准测试
OptionWorld: A Benchmark for Diagnosing Failures of LLMs as Long-Term Personal Assistants
AI 导读
研究者推出 OptionWorld 基准，通过模拟日常任务追踪信息获取、判断、行动与结果，定位 LLM 作为长期个人助理的失败环节。在归档决策输入的运行中，35.5% 的失败实例里关键事实仍存在于模型截止前的最后一次输入中。失败分析显示三类最常见问题：所需信息未被获取或整合、可用的行动机会未被使用、智能体施加了任务并不要求的限制。
正文 · 原文
Abstract:Long-term assistants must follow unfinished tasks across days as information and circumstances change. Finding information or starting an action, however, does not ensure that the task gets completed. We introduce OptionWorld, a benchmark that traces information acquisition, judgments, actions, and outcomes in simulated daily-life tasks to identify where current LLMs fall short as long-term personal assistants. In runs with archived decision inputs, the key facts remain in the model's last input before the deadline in 35.5% of failed instances. Failure analysis identifies the three most frequent problems: needed information is not obtained or combined, available opportunities to act go unused, and agents impose restrictions that the task does not require. Three follow-up experiments show that supplying missing facts restores task success in some cases; action reminders can restart work while subsequent steps still miss the deadline; and correcting a supplied note that wrongly forbids contact restores permitted inquiry. These findings locate shortcomings in maintaining task state: connecting facts to unfinished work, following through before deadlines, and separating actual restrictions from the agent's assumptions.
| Comments: | 31 pages, 3 figures |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.28491 [cs.CL] |
| (or arXiv:2610.28491v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.28491 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ruoyu Zhang [view email]
[v1]
Tue, 20 Oct 2026 16:38:43 UTC (848 KB)
来源：arXiv cs.CL · arxiv.org