arXiv cs.AI· Bowen Yang, Kaiming Jin, Zichen Ding, Zhenyu Wu, Zehao Li, Zhoumianze Liu, Kanzhi Cheng, Fangzhi Xu, Yanan Sun, Qiushi Sun, Yu Qiao·· 2 天前AI 评分52
OS-Helix：在原生计算机智能体中统一 GUI 交互与代码执行
OS-Helix: Unifying GUI Interaction and Code Execution in Native Computer-Using Agents
AI 导读
OS-Helix 是一个端到端训练的多模态计算机使用智能体，把 GUI 操作与代码执行统一到同一个动作空间。作者通过应用与文件产物配对构建可执行 Ubuntu 状态，并用解耦的指令与评估器生成流程产出可验证任务，先以 SFT 冷启动、再在真实桌面环境中做在线 RL，并提出按动作步聚合重要性权重的 Step-balanced GSPO。
正文 · 原文
Authors:Bowen Yang, Kaiming Jin, Zichen Ding, Zhenyu Wu, Zehao Li, Zhoumianze Liu, Kanzhi Cheng, Fangzhi Xu, Yanan Sun, Qiushi Sun, Yu Qiao
Abstract:Real desktop automation is neither purely visual nor purely programmatic. Agents must understand user-facing interfaces while efficiently manipulating files, scripts, and system state. However, native GUI/code coordination cannot be obtained by prompt-level tool composition. In our preliminary experiments, naively adding a code tool degrades a GUI agent, revealing that GUI/code routing must be internalized through training on high-quality hybrid data. We present OS-Helix, an end-to-end trained multimodal computer-using agent that unifies GUI actions and code execution in one action space. To construct the required supervision, we introduce application-artifact composition, pairing desktop applications with authentic file artifacts to instantiate executable Ubuntu states, and a decoupled instruction/evaluator generation pipeline that produces environment-grounded tasks whose final states can be verified regardless of the solution modality. We cold-start the policy with SFT on filtered hybrid data and further improve it through online RL in live desktop environments. For RL, we propose Step-balanced GSPO, which aggregates importance weight by action step rather than raw assistant tokens to prevent long code actions from dominating heterogeneous-action optimization. On OSWorld-Verified, OS-Helix-27B and OS-Helix-35B-A3B achieve relative improvements of 35.0% and 33.7% over their matched Qwen3.5 hybrid baselines, while OS-Helix-27B-RL further improves by 42.9% to reach 66.44% success. Ablations provide evidence consistent with contributions from learned GUI/code coordination, scalable verifiable supervision, and action step balanced optimization, while demonstrating cross-platform transfer on WindowsAgentArena. Our project homepage is at this https URL.
| Comments: | Project page: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2610.22234 [cs.LG] |
| (or arXiv:2610.22234v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22234 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Bowen Yang [view email]
[v1]
Thu, 15 Oct 2026 17:14:02 UTC (3,842 KB)
来源：arXiv cs.AI · arxiv.org