arXiv cs.CL· Monica Riedler, Samuele Antonelli, Barbara Plank, Raffaella Bernardi·· 2 天前AI 评分31
AmbiSteer:在视觉语言模型中选择性激发歧义感知响应
AmbiSteer: Selectively Eliciting Ambiguity-Aware Responses in Vision-Language Models
AI 导读
AmbiSteer 是一个轻量框架,通过激活引导选择性地激发视觉语言模型的歧义感知行为。研究发现单独引导效果有限且依赖模型,提示词更有效,但二者结合会在无歧义问题上误触发澄清或枚举行为。AmbiSteer 用线性分类器预测输入歧义并据此调节引导方向,使模型正确处理输入的比例从 59.0% 提升至 77.0%。
正文 · 原文
Abstract:Vision-language models (VLMs) often exhibit overconfidence, providing firm answers even when visual evidence is insufficient. A prominent case is referential ambiguity in Visual Question Answering (VQA), where multiple entities in an image are plausible referents of a question. Addressing such cases requires recognizing when ambiguity is present: ambiguous questions can be handled by asking for clarification or enumerating plausible referents, while unambiguous questions should receive a direct answer. Current VLMs struggle with this distinction, often defaulting to a single interpretation. To investigate whether activation steering can serve as an inference-time mechanism to elicit such behaviors, we introduce AmbiSteer, a lightweight framework to selectively elicit ambiguity-aware behavior. Across three VLMs, we find that steering alone has limited and model-dependent effects, while prompting is generally more effective. Combining prompting and steering substantially increases clarification and enumeration behaviors, but also elicits them on unambiguous cases. AmbiSteer addresses this using a linear classifier to predict input ambiguity and condition the steering direction accordingly, increasing the proportion of inputs handled appropriately from 59.0% to 77.0% on average across models. Our work sets the ground for lightweight approaches to reduce model overconfidence in uncertain multimodal settings.
| Comments: | 23 pages, 23 figures |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.17900 [cs.CV] |
| (or arXiv:2611.17900v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.17900 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Monica Riedler [view email]
[v1]
Fri, 13 Nov 2026 08:40:07 UTC (7,610 KB)
来源:arXiv cs.CL · arxiv.org