arXiv cs.AI· Sivasuthan Sukumar, Uthayasanker Thayasivam·· 1 天前AI 评分26
基于熵的视图选择看不到什么：视觉语言模型测试时适应的审计研究
What Entropy-Based View Selection Cannot See: An Audit of Test-Time Adaptation for Vision-Language Models
AI 导读
研究发现，视觉语言模型测试时适应中，视图选择而非梯度更新贡献了几乎全部准确率增益；但当预测错误时，基于熵的视图选择排序正确视图的能力与随机猜测相当，仅恢复oracle约二十分之一的准确率。该局限在两种图像编码器下均成立，原因是熵只衡量置信度而不反映视图支持哪个类别。作者利用增强流程中已有的信号——每个裁剪保留原图的比例——进行排序，在15个基准数据集中的14个上优于熵方法。
正文 · 原文
Abstract:Vision-language models such as CLIP can classify images into categories they were never explicitly trained to recognize, but accuracy degrades when test images differ in style or source from their training data. A common remedy generates randomly cropped and flipped copies of each test image, called views, and combines their predictions. Since some views miss the object, methods must identify which views to trust. The dominant approach uses prediction entropy to measure confidence and retains the most certain views. Other methods use signals such as embedding typicality. Despite repeated refinements of these scoring functions, the information they rely on, the model's own response to each view, remains largely unexamined. We investigate this choice while holding other pipeline stages fixed. First, we show that view selection, rather than the gradient update, accounts for almost all accuracy gains. We then examine whether entropy identifies useful views when predictions are wrong, despite at least one view predicting the correct class. In these cases, entropy ranks correct views above incorrect ones no better than chance, recovering only about one-twentieth of the accuracy recovered by an oracle over the same views. This finding holds across two image encoders and under a control ruling out an obvious statistical artifact. An exact analysis explains this limitation: entropy measures confidence but not which class a view favors. We identify an unused signal already available in the augmentation pipeline: the fraction of the original image retained by each crop. Ranking views by this quantity improves accuracy on 14 of 15 benchmark datasets over entropy and on 11 of 15 over the strongest scoring function tested. A pre-method measurement also predicts which datasets benefit.
| Comments: | Accepted at NeurIPS 2026 Workshops on VLM4RWD |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.14686 [cs.CV] |
| (or arXiv:2611.14686v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14686 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sivasuthan Sukumar [view email]
[v1]
Tue, 10 Nov 2026 19:45:46 UTC (121 KB)
来源：arXiv cs.AI · arxiv.org