arXiv cs.AI· Ruilin Yao, Xinyu Qiu, Tianyu Zou, Yi Rong, Shengwu Xiong, Shuheng Shen, Changhua Meng·· 13 小时前AI 评分42
LBS：从 GUI grounding 学习可迁移的主动视觉观察能力
Learning to Look Beyond Screens: Transferable Visual Observation from GUI Grounding
AI 导读
研究者提出 LBS（Look Beyond Screens），先以教师生成的 GUI grounding 轨迹做监督初始化，再用强化学习训练多轮裁剪-检查决策。
正文 · 原文
Abstract:Visual perception is evolving from passive recognition toward active, task-oriented observation. We investigate GUI grounding as a scalable source of spatial supervision for learning observation behaviors that transfer beyond interfaces. We introduce LBS (Look Beyond Screens), a vision-language model trained through supervised initialization on teacher-generated GUI grounding trajectories, followed by reinforcement learning for multi-turn crop-and-inspect decisions. Under fixed inference settings, LBS models based on Qwen3.5-35B-A3B and Qwen3.8-27B consistently improve over their corresponding baselines. Gains beyond supervised fine-tuning transfer to PerceptionBench for both backbones and to HR-Bench and ChartQAPro for Qwen3.5-35B-A3B, supporting observation strategies that generalize beyond GUI-specific imitation. Further analysis suggests complementary roles for the two stages: SFT establishes visual inspection primitives, while RL improves their downstream effectiveness. These findings support GUI grounding as a practical supervision paradigm for transferable active visual perception and broader multimodal reasoning. Our code and model are publicly available at this https URL and this https URL, respectively.
| Comments: | Technical Report |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.37093 [cs.CV] |
| (or arXiv:2610.37093v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37093 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ruilin Yao [view email]
[v1]
Tue, 27 Oct 2026 11:56:00 UTC (6,751 KB)
来源：arXiv cs.AI · arxiv.org