arXiv cs.AI· Xing Han L\`u, Dheeraj Vattikonda, Sina Hajimiri, Fatemeh Pesaran Zadeh, Parishad BehnamGhader, Ghazwa Darwiche, Amirhossein Kazemnejad, Christopher Pal, Alexandre Drouin, Siva Reddy·· 9 天前AI 评分26
AgentHorizon：评估长周期计算机使用任务的智能体评判器
AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use Tasks
AI 导读
研究者推出 AgentHorizon 基准，包含 1,373 个计算机使用任务（指令-轨迹对），来自 166 小时人类录制的轨迹，覆盖三种操作系统。该基准通过交换相近指令构造负样本任务，评估评判器区分成功轨迹与完成相似但不兼容请求的能力，并发布 AH、AH-S、AH-D 三个划分。
正文 · 原文
Abstract:Computer-use agents are capable of completing complex tasks, increasing the use of automatic judges to determine success, either for training or for evaluation without human involvement. Despite their flexibility, their reliability on long tasks spanning multiple applications remains unclear. A trajectory, composed of long sequences of screenshots and actions, may appear complete, but in reality violates constraints from the instruction or introduces an unwanted side effect. To identify these errors, a judge needs to examine the trajectory with respect to the user's instruction. To this end, we introduce AgentHorizon, a benchmark of 1,373 computer-use tasks (instruction-trajectory pairs) drawn from 166 hours of human-recorded trajectories spanning three operating systems. By recording trajectories for closely related instructions, we can construct negative tasks by swapping the instructions. This paired design evaluates judges on their ability to distinguish a successful trajectory from one that completed a similar (but incompatible) request. We release the benchmark under three splits: a frontier split, AgentHorizon (AH), a simplified split, AgentHorizon-Simple (AH-S), and a development split, AgentHorizon-Development (AH-D). We further evaluate eleven judges by (1) directly passing the full trajectory (with up to 300 screenshots and actions), and (2) using them as coding agents across five agent harnesses. We find that our best agentic judge, GPT-5.5, achieves 80.9% balanced accuracy on the AH subset. We find that tool-use improves certain models but results in worse performance for open-weight models, and that judges differ drastically in their ability to accept a valid trajectory and reject failed ones. Our findings highlight the need for judges that are capable of locating and verifying often hidden evidence that a task was properly completed inside long interaction histories.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.11050 [cs.AI] |
| (or arXiv:2610.11050v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11050 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xing Han Lù [view email]
[v1]
Thu, 8 Oct 2026 01:03:48 UTC (2,705 KB)
来源：arXiv cs.AI · arxiv.org