面向 GUI grounding 的质量感知自蒸馏方法

View PDF HTML (experimental)

Abstract:GUI grounding requires vision-language models (VLMs) to identify small target elements in high-resolution screenshots and predict precise screen coordinates. On-policy self-distillation (OPSD) is a promising post-training approach for this coordinate-sensitive task because it provides dense token-level teacher signals beyond hard coordinate labels. However, naive OPSD is not well-suited to GUI grounding: since the teacher is evaluated on student-generated prefixes, its coordinate-token signals can become unreliable once a prefix deviates from the target coordinate. We propose quality-aware self-distillation for GUI grounding, centered on direct spatial correctness verification of teacher predictions. A soft gate checks whether the teacher's top coordinate-token prediction can still be completed into the ground-truth box under the student-generated prefix and down-weights the signal when this check fails. Teacher-probability scaling then uses the teacher's confidence to further calibrate the gated supervision. A key empirical finding is that neither component alone improves overall performance, whereas combining them consistently does. This suggests that the two mechanisms play complementary roles: gating suppresses unreliable coordinate-token supervision, while probability scaling calibrates the remaining signals. Experiments across six GUI grounding benchmarks show that our method consistently improves the base model and outperforms strong baselines.

Submission history

From: Jingyuan Huang [view email]
[v1] Tue, 16 Jun 2026 16:02:41 UTC (3,732 KB)
[v2] Wed, 17 Jun 2026 08:58:52 UTC (3,732 KB)
[v3] Fri, 23 Oct 2026 02:03:47 UTC (523 KB)