arXiv cs.AI· Zhaoyang Chu, Jiarui Hu, Xingyu Jiang, Pengyu Zou, Han Li, Chao Peng, Peter O'Hearn, Earl T. Barr, Mark Harman, Federica Sarro, He Ye·· 12 小时前AI 评分47
TerminalWorld：在真实终端任务上评测 AI 智能体的基准
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
AI 导读
TerminalWorld 是一个可扩展的数据引擎，能从真实终端录屏中自动逆向生成高保真评测任务，处理 80,870 条录屏后产出 1,530 个经校验任务，覆盖 18 个真实类别、1,280 个唯一命令，并从中人工审核出 200 个任务的 Verified 子集。
正文 · 原文
Authors:Zhaoyang Chu, Jiarui Hu, Xingyu Jiang, Pengyu Zou, Han Li, Chao Peng, Peter O'Hearn, Earl T. Barr, Mark Harman, Federica Sarro, He Ye
Abstract:We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings. Processing 80,870 terminal recordings, the engine yields a full benchmark of 1,530 validated tasks, spanning 18 real-world categories, ranging from short everyday operations to workflows exceeding 50 steps, and covering 1,280 unique commands. From these, we curate a Verified subset of 200 diverse, manually reviewed tasks. Comprehensive benchmarking on TerminalWorld-Verified across eight frontier models and six agents reveals that current systems still struggle with authentic terminal workflows, achieving a maximum pass rate of only 62.5%. Moreover, TerminalWorld captures real-world terminal capabilities distinct from existing expert-curated benchmarks (e.g., Terminal-Bench), with only a weak correlation to their scores (Pearson r=0.20). The automated engine makes TerminalWorld authentic and scalable by construction, enabling it to evaluate agents in real-world terminal environments as developer practices evolve.
| Comments: | Accepted at NeurIPS 2026 Evaluations and Datasets Track. Project website: this https URL |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.22535 [cs.AI] |
| (or arXiv:2605.22535v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.22535 arXiv-issued DOI via DataCite |
Submission history
From: Zhaoyang Chu [view email]
[v1]
Thu, 21 May 2026 14:24:43 UTC (715 KB)
[v2]
Wed, 4 Nov 2026 23:26:30 UTC (721 KB)
来源：arXiv cs.AI · arxiv.org