arXiv cs.CL· Hongxing Li, Dingming Li, Yixin Li, Yong Du, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen·· 9 天前AI 评分30
ViSkill：用可演进的视觉原生技能强化 VLM 智能体
ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skills
AI 导读
ViSkill 是一个视觉原生技能学习框架，将成功交互编码为 VLM 智能体可直接读取的复合视觉技能卡，检索到的技能同时引导推理与奖励塑形，成功轨迹再蒸馏回技能库形成闭环。在 Sokoban、FrozenLake 和 PrimitiveSkill 上，ViSkill 整体成功率达 0.89，加入冷启动初始化后升至 0.91，优于所有评测的闭源与开源基线，且收敛快于标准 PPO。
正文 · 原文
Abstract:Skill-augmented agents improve sample efficiency by distilling successful trajectories into reusable strategies. Yet most existing approaches remain text-centric, linearizing spatial layouts and action-state correspondences into language that loses critical geometric structure. Recent efforts have begun incorporating visual evidence, but construct and update skills separately from policy optimization, leaving their mutual improvement underexplored. We propose ViSkill, a visual-native skill learning framework that encodes successful interactions as composite visual skill cards directly accessible to VLM agents. Retrieved skills guide both inference and reward shaping, while successful trajectories are distilled back into the library, forming a closed feedback loop in which skill accumulation and policy improvement reinforce each other. An optional cold-start mechanism further accelerates early-stage learning. Evaluated on Sokoban, FrozenLake, and PrimitiveSkill, ViSkill achieves an overall success rate of 0.89, rising to 0.91 with cold-start initialization, outperforming all evaluated proprietary and open-source baselines while converging faster than standard PPO. Our code is available at this https URL.
| Comments: | Code: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.12403 [cs.CV] |
| (or arXiv:2610.12403v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12403 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hongxing Li [view email]
[v1]
Thu, 8 Oct 2026 17:44:19 UTC (1,028 KB)
来源：arXiv cs.CL · arxiv.org