arXiv cs.AI· Alice Gao, Weixi Tong, Rishab Vempati, Katharina Reinecke, R. Benjamin Shapiro, Tianyi Zhang, Jason Wu·· 1 天前AI 评分31
通过界面变体相对比较训练 Computer Use Agents 评估 GUI 可用性
Training Computer Use Agents to Assess the Usability of Graphical User Interfaces
AI 导读
研究提出一种将可用性操作化的训练目标，通过界面变体间的相对比较微调 Computer Use Agents（CUA）来评估 GUI 可用性，训练数据为大规模完全交互式 UI 与可用性标签及人类数据混合。在真实与合成 GUI 上的实证评估显示，该方法提升了三个现成 CUA 模型的相对可用性评估准确率，最佳模型超越专有基线，并产出有依据的可用性评估。
正文 · 原文
Abstract:Recent HCI research leverages AI agents to assess graphical user interfaces (GUIs) usability, but current agents still struggle. Computer use agents (CUAs) that are used in these assessments are predominantly developed using objective task trajectories with verifiable outcomes and lack an operationalized definition of usability and necessary training data. This work presents a novel training objective that operationalizes usability to fine-tune CUAs to assess GUI usability through relative comparisons between interface variants. We apply our method to CUA models using a large-scale dataset of fully interactive UIs paired with usability labels mixed with human this http URL empirical evaluations across real and synthetic GUIs show that our method i) improved relative usability assessment accuracy across three off-the-shelf CUA models, ii) enabled our best model to outperform proprietary baselines, and iii) produced grounded usability assessments. Our work lays a data-driven foundation for agent-based usability assessment, enabling exploration of new forms of scalable usability evaluation and interface design.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2604.26020 [cs.CL] |
| (or arXiv:2604.26020v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.26020 arXiv-issued DOI via DataCite |
Submission history
From: Alice Gao [view email]
[v1]
Tue, 28 Apr 2026 18:04:11 UTC (5,812 KB)
[v2]
Tue, 10 Nov 2026 20:28:07 UTC (6,457 KB)
来源：arXiv cs.AI · arxiv.org