arXiv cs.AI· Yubo Cui, Xianchao Guan, Zheng Zhang·· 10 小时前AI 评分25
TRAG：面向视觉语言模型零样本对抗鲁棒性的文本响应感知引导方法
Textual Response-Aware Guidance for Zero-Shot Adversarial Robustness of Vision-Language Models
AI 导读
研究者提出文本响应感知引导（TRAG），利用预训练文本编码器对输入嵌入局部变化的响应方向与幅度构建锚点级约束，并按预训练图文相关性聚合为图像自适应约束，以抑制视觉表征漂移。跨架构与任务实验显示，TRAG 在干净与鲁棒性能上均达到 SOTA，并有助于保持预训练图文关系。
正文 · 原文
Abstract:Vision-language models (VLMs) exhibit strong zero-shot generalization but remain vulnerable to adversarial attacks. Existing adversarial fine-tuning methods often use pre-trained model guidance to preserve image--text alignment while improving robustness. Our analysis of visual representation drift reveals complementary limitations of two representative forms of guidance. Matching visual features constrains all directions equally, providing broad coverage without semantic emphasis. Matching prediction distributions prioritizes semantically relevant directions, but remains limited by the candidate texts and prediction probabilities. These limitations can hinder preservation of pre-trained image--text relationships and degrade zero-shot performance. We therefore propose Textual Response-Aware Guidance (TRAG), which constrains visual representation drift using the pre-trained text encoder's responses to local variations in input embeddings. TRAG constructs anchor-level constraints from the response directions and magnitudes around semantic anchors, then aggregates them using pre-trained image--anchor relevance, yielding an image-adaptive constraint. Extensive experiments across architectures and tasks demonstrate state-of-the-art clean and robust performance. Further analysis shows that TRAG helps preserve pre-trained image--text relationships.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multimedia (cs.MM) |
| Cite as: | arXiv:2610.40309 [cs.CV] |
| (or arXiv:2610.40309v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.40309 arXiv-issued DOI via DataCite |
Submission history
From: Yubo Cui [view email]
[v1]
Thu, 29 Oct 2026 15:48:43 UTC (459 KB)
[v2]
Mon, 9 Nov 2026 17:35:03 UTC (1,394 KB)
来源：arXiv cs.AI · arxiv.org