arXiv cs.AI· Yuqi Huang, Hui Li, Yifei Cao, Zikun Cai, Tianci Xie, Yangfan Gao, Pinxue Guo, Chaoyun Zhang, Bingxuan Li, Rui Zheng, Hang Yan, Ning Liao·· 6 天前AI 评分41
Work2Skill：从人类 GUI 经验中归纳可复用智能体技能
Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction
AI 导读
Work2Skill 框架通过 WorkRecorder 采集屏幕录制、鼠标键盘事件与无障碍树信息，构建了首个同步多模态人类桌面 GUI 演示数据集 WorkDemo-350（350 条演示），并提出按需检查证据的智能体技能归纳方法，生成包含工作流规则、步骤依赖与完成校验的可复用技能。
正文 · 原文
Authors:Yuqi Huang, Hui Li, Yifei Cao, Zikun Cai, Tianci Xie, Yangfan Gao, Pinxue Guo, Chaoyun Zhang, Bingxuan Li, Rui Zheng, Hang Yan, Ning Liao
Abstract:Computer-use agents can perform increasingly complex GUI tasks, yet unfamiliar software and workflows requiring human domain knowledge remain challenging. Human demonstrations convey procedural knowledge through the same GUI interaction space, but reuse requires interpreting actions in context and separating workflow rules from task-specific details. We thus introduce Work2Skill, a framework that captures human GUI experience for reusable agent skill induction. WorkRecorder provides the framework's infrastructure for collecting this experience, aligning screen recordings, mouse and keyboard events, and available accessibility-tree information. Using WorkRecorder, we construct WorkDemo-350, the first synchronized multimodal dataset of human desktop GUI demonstrations for reusable agent skill induction and downstream evaluation. Its 350 demonstrations are screened for completeness and task success. We then propose an agentic skill-induction method that inspects evidence on demand to produce reusable skills capturing workflow rules, step dependencies, and completion checks. When the same four execution models are equipped with skills induced by each method under shared execution settings, Work2Skill achieves the highest scores and success rates. Mean score improvements over the corresponding No Skill baseline are 4.67--10.54 points on MyPCBench Bounded Action and 3.11--7.50 points on a 286-task subset of OSWorld-Verified. Our code and data are available at this https URL.
| Subjects: | Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC) |
| Cite as: | arXiv:2610.12984 [cs.AI] |
| (or arXiv:2610.12984v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12984 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuqi Huang [view email]
[v1]
Fri, 9 Oct 2026 04:32:02 UTC (3,848 KB)
来源：arXiv cs.AI · arxiv.org