arXiv cs.AI· Hanqi Mao, Jiajun Fan, Ruihan Guo, Shuaike Shen, Ziwen Wang, Hui Ren, Xianrui Zhong, Ge Liu·· 10 小时前AI 评分38
Vanguard:视觉语言智能体推测执行中如何判断草稿可信
Vanguard: Knowing When to Trust the Draft in Speculative Vision-Language Agents
AI 导读
Vanguard 是一个免训练的判断器,在视觉语言智能体的步骤级推测执行中结合截图评估小模型草稿是否正确,通过阈值决定保留草稿还是交由大模型重写。在 web、移动与具身智能体任务上,2B 草稿经 Vanguard 判断后准确率不低于 8B 目标模型,在 Mind2Web 和 EB-ALFRED 上显著更高,且目标模型只负责少数步骤。
正文 · 原文
Abstract:Vision-language agents that browse websites, operate phones and act in homes reason before every action, so every step waits for a large model, a latency and cost that stand in the way of deployment. Step-level speculation lets a small model draft each step and calls the large model only on rejected drafts, but its outcome rests on the judge, and a judge that reads only a step's text cannot tell whether the step fits the screen it acts on. We find that a few samples of a small model can contain a correct step about as often as the large model gets the step right, and not always on the same steps: checked against the screen, small drafts can be not only cheaper but better. We present Vanguard, a training-free judge that leaves the draft and target models unchanged: a general decision model, with the screenshot in view, gives each candidate step a probability of being right in one forward pass; the best candidate is kept if it clears a threshold, and otherwise the large model writes the step. On web, mobile and embodied agents, a 2B draft judged by Vanguard is at least as accurate as its 8B target and significantly more accurate on Mind2Web and EB-ALFRED, while the target writes only a minority of the steps; it completes far more closed-loop episodes, and with a 27B target from another family it is both faster and more accurate. Vanguard is the most accurate judge we compare on the agent tasks; it relies on the screenshot on web and mobile tasks, its calibrated threshold transfers to unseen websites, and across model families its gain over the target grows with the draft's pass@4 relative to the target's accuracy. Vanguard composes with token-level speculative decoding and matches trained process reward models on mathematics, turning speculation from a way to save the large model's work into one that can improve on it.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23917 [cs.AI] |
| (or arXiv:2611.23917v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23917 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hanqi Mao [view email]
[v1]
Wed, 18 Nov 2026 06:09:08 UTC (11,821 KB)
来源:arXiv cs.AI · arxiv.org