arXiv cs.AI· Shiqi Zhang, Guochen Zhou, Zesheng Wei, Hai Zhao, Zhisong Zhang·· 14 小时前AI 评分35
DynaRev：评估动态目标修订下的 Web Agent 基准
DynaRev: Benchmarking Web Agents under Dynamic Goal Revision
AI 导读
DynaRev 是一个包含 181 个任务的基准，用于评估 Web Agent 在任务执行过程中面对用户动态修订目标时的表现。它通过语义里程碑在任务既定阶段下发修订，覆盖增量修订、无条件覆盖和运行时条件覆盖三类，并从正确性、响应性和效率三方面评估。五模型评测显示 Agent 往往只满足部分任务要求，保留浏览器状态并重写指令可带来收益，而重置浏览器则造成损失。
正文 · 原文
Abstract:Web agents are typically evaluated with goals fully specified before execution and fixed thereafter, whereas real users often revise their requirements during interaction. To study such interactions, we introduce DynaRev, a benchmark of 181 tasks to evaluate web agents under dynamic goal revision. DynaRev uses semantic milestones to deliver revisions at task-defined stages, covers additive revisions, unconditional overrides, and runtime-conditioned overrides, and evaluates correctness, responsiveness, and efficiency through trajectory-grounded evaluation. Five-model evaluations show that agents often satisfy only part of the task requirements, with performance varying across task sources, revision types, and models. Further analyses reveal varied milestone timing, mixed effects of dynamic delivery, and contrasting revision-handling outcomes, with gains from instruction rewriting while preserving browser state and losses from browser resets. DynaRev provides a testbed for studying how web agents adapt to evolving user goals. The code and data are available at this https URL.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.40133 [cs.AI] |
| (or arXiv:2610.40133v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.40133 arXiv-issued DOI via DataCite |
Submission history
From: Shiqi Zhang [view email]
[v1]
Thu, 29 Oct 2026 14:25:54 UTC (972 KB)
来源：arXiv cs.AI · arxiv.org