OS-Wizard：用知识引导探索将电脑操作智能体扩展到专用软件

Authors:Zhoumianze Liu, Zichen Ding, Bowen Yang, Zehao Li, Zhenyu Wu, Yibo Zhao, Kanzhi Chen, Yian Wang, Yanan Sun, Qiushi Sun, Yu Qiao

View PDF HTML (experimental)

Abstract:Computer-using agents are increasingly capable of performing complex tasks in the digital world. However, high-quality interaction trajectories remain scarce for specialized desktop applications, where limited application knowledge, cumbersome GUI workflows, and incomplete exploration hinder automated collection. We introduce OS-Wizard, an automated trajectory collection framework. It combines tutorial-guided exploration with a hybrid GUI--Code action space, providing application-specific knowledge and simplifying GUI workflows through direct execution of application-native commands. The framework also checkpoints and revisits pivotal states during tree exploration to explore alternative workflows, complementing linear rollouts for broader coverage. We collect 10.9K trajectories comprising 54.9K state--action samples. Fine-tuning open-source base models on this corpus yields a best-performing model that improves performance on a benchmark of specialized desktop workflows by 58.2 percentage points to 64.1\%, surpassing reported human performance of 59.8\% and establishing a new state of the art. The fine-tuned model also improves performance on general desktop workflows using GUI actions only, suggesting that the collected supervision benefits general desktop tasks as well. Comparative experiments and ablations demonstrate the efficiency of trajectory collection and the importance of code actions.

Subjects:	Artificial Intelligence (cs.AI)
Cite as:	arXiv:2610.33907 [cs.AI]
	(or arXiv:2610.33907v1 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2610.33907 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Zhoumianze Liu [view email]
[v1] Sat, 24 Oct 2026 20:39:23 UTC (1,023 KB)