arXiv cs.CL· Chengguang Gan, Yunhao Liang, Hanjun Wei, Qinghao Zhang, Zhixi Cai, Shiwen Ni·· 3 天前AI 评分31
为什么还要生成点击？面向 Web 智能体的插件式决策头
Why Generate a Click? Plug-in Decision Heads for Web Agents
AI 导读
研究者提出插件式决策头，直接从冻结的视觉语言模型生成器隐藏状态中读取动作类型、目标元素和文本选项，无需生成 click(12) 这类文本。在 Qwen3-VL 4B 与 8B 上基于 213k 网页步骤训练后，置信级联在七个离线测试集上与生成器差距在 -0.3 至 +1.4 分之间，决策头承担 12% 至 64% 的步骤。
正文 · 原文
Abstract:Web agents built on vision language models decide every action by generating text such as click(12), although most of that text encodes a choice among a few action types and a few dozen marked elements. We read these choices directly from the hidden states that the generator computes while it reads the goal and the screenshot. Plug-in decision heads on a frozen generator predict the action type, pool the visual tokens inside each numbered box to pick the target, and point at a listed text option or at a WRITE slot that leaves the writing of new text to the generator. A confidence cascade executes the heads' action when they are sure and otherwise lets the generator decode from the KV cache of the same forward pass. With Qwen3-VL 4B and 8B generators trained on 213k web steps, a strict threshold keeps the cascade between 0.3 points below and 1.4 points above the generator on seven offline test sets while the heads decide 12 to 64% of the steps. A looser threshold raises step success on unseen Mind2Web websites by 3.1 and 1.8 points, costs up to 2 points on GUIAct and MiniWoB++, and lifts WebArena task success from 12.5 to 13.8% (4B) and from 13.6 to 14.6% (8B) with less than half of the generated tokens. Training the heads jointly with the generator lowers the generator's own step success by up to 4.9 points; heads on a frozen generator avoid this cost.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.20107 [cs.CL] |
| (or arXiv:2610.20107v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20107 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Chengguang Gan [view email]
[v1]
Wed, 14 Oct 2026 12:39:51 UTC (479 KB)
来源：arXiv cs.CL · arxiv.org