arXiv cs.AI· Guangzhi Wang, Junfu Pu, Yiran Wang, Xingyilang Yin, Honghao Chen, Cheng Cheng, Lingen Li, Yinan Zhou, Jiarui Wu, Yinuo Jiang, Hongbo Jin, Xiaodong Cun, Ying Shan·· 14 小时前AI 评分31
TaskFrame：连接任务级规划与帧级控制的游戏智能体框架
TaskFrame: Connecting Task-Level Planning with Frame-Level Control for Game Agents
AI 导读
TaskFrame 是一个分层规划器-控制器框架，将任务级规划与帧级键鼠控制连接起来。视觉语言任务规划器把高层目标分解为原子指令序列，经监督学习和依赖感知 GRPO 优化，其目标 token 隐藏状态作为可缓存的连续目标嵌入供控制步骤复用；因果 Transformer 控制器结合目标嵌入与近期观测，逐步预测离散键鼠动作和连续镜头移动。
正文 · 原文
Authors:Guangzhi Wang, Junfu Pu, Yiran Wang, Xingyilang Yin, Honghao Chen, Cheng Cheng, Lingen Li, Yinan Zhou, Jiarui Wu, Yinuo Jiang, Hongbo Jin, Xiaodong Cun, Ying Shan
Abstract:Building general game agents requires both high-level task reasoning and low-latency action control in rapidly changing visual environments. These capabilities favor models with substantially different computational profiles, making them difficult to realize within a single model. In this work, we present TaskFrame, a hierarchical planner-controller framework that connects task-level planning with frame-level keyboard-and-mouse control. A vision-language task planner decomposes a high-level goal into an ordered sequence of atomic instructions and is optimized through supervised learning and dependency-aware GRPO. The hidden states of special goal tokens that terminate each atomic instruction provide continuous goal embeddings that are cached and reused across control steps. A causal Transformer controller combines the embedding of the active goal with recent observations to predict discrete keyboard and mouse-button actions and continuous camera motion at every game step. We train the system in three stages: planner training, controller pretraining, and joint alignment. Experiments in Minecraft show that TaskFrame is competitive with general-purpose vision-language models on instruction decomposition, and that in closed-loop evaluation the agent achieves a higher combat success rate than the evaluated baselines.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.40276 [cs.CV] |
| (or arXiv:2610.40276v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.40276 arXiv-issued DOI via DataCite |
Submission history
From: Guangzhi Wang [view email]
[v1]
Thu, 29 Oct 2026 15:31:37 UTC (1,826 KB)
来源：arXiv cs.AI · arxiv.org