arXiv cs.AI· Zichuan Fu, Shirong Wang, Wenlin Zhang, Guojing Li, Yimin Deng, Jingtong Gao, Junjia Qi, Hanyu Yan, Yefeng Zheng, Xiaopeng Li, Wanyu Wang, Xian Wu, Xiangyu Zhao·· 15 小时前AI 评分40
GUI-Lens：通过由粗到细裁剪让现成 VLM 实现精准 GUI 定位
GUI-Lens: Enabling Precise GUI Grounding for Off-the-Shelf VLMs via Coarse-to-Fine Cropping
AI 导读
GUI-Lens 是一个由粗到细的 GUI 定位框架，让通用 VLM 通过主动视觉观察逐步确定点击目标，在 ScreenSpot-Pro 上定位准确率最高达直接定位的 9.3 倍。
正文 · 原文
Authors:Zichuan Fu, Shirong Wang, Wenlin Zhang, Guojing Li, Yimin Deng, Jingtong Gao, Junjia Qi, Hanyu Yan, Yefeng Zheng, Xiaopeng Li, Wanyu Wang, Xian Wu, Xiangyu Zhao
Abstract:GUI grounding maps natural-language instructions to click locations and is essential for reliable GUI agents. The task remains difficult on high-resolution, densely populated interfaces because a vision-language model (VLM) may recognize a requested control without locating it precisely enough for interaction. Most existing methods provide various forms of localization assistance, but still rely on a direct click prediction, allowing visual ambiguity or an inaccurate initial estimate to propagate to the final result. In this paper, we introduce GUI-Lens, a coarse-to-fine grounding framework that allows a general-purpose VLM to determine the target through active visual observations. Specifically, GUI-Lens extracts OCR text and detected UI components from the screenshot and presents their positions as coordinate references. Using the instruction, the current view, and these references, the VLM selects the region and scale of the next view, which is cropped and enlarged to provide finer visual details. This process continues over successively focused views until the target is determined. Proposed crops and clicks are checked against the instruction throughout the process, and the final local position is mapped back to the original screen coordinates. Experiments on five GUI grounding benchmarks and six general-purpose base models show that GUI-Lens consistently improves grounding accuracy, achieving up to 9.3 times the accuracy of direct grounding on ScreenSpot-Pro.
| Comments: | Preprint. Code: this https URL ; Project page: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.03270 [cs.CV] |
| (or arXiv:2608.03270v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2608.03270 arXiv-issued DOI via DataCite |
Submission history
From: Shirong Wang [view email]
[v1]
Tue, 4 Aug 2026 07:47:37 UTC (714 KB)
[v2]
Fri, 30 Oct 2026 05:45:06 UTC (763 KB)
来源：arXiv cs.AI · arxiv.org