arXiv cs.AI· Jan Rathjens, Shirin Reyhanian, David Kappel, Laurenz Wiskott·· 6 天前AI 评分24
通过逆向网络理解基于 Transformer 的视觉模型
Understanding Transformer-based Vision Models through Inversion
AI 导读
研究提出一种模块化的特征逆向方法，可从中间表征重建图像，计算效率更高且部分情况下语义更连贯。该方法应用于 Detection Transformer、Vision Transformer。
正文 · 原文
Abstract:Understanding the internal mechanisms of deep neural networks remains a central challenge in machine learning. In computer vision, one promising yet only preliminarily explored approach is feature inversion via inverse networks, which reconstructs images from intermediate representations using trained inverse networks. In this study, we revisit feature inversion via inverse networks, introducing a novel, modular variant that enables a computationally more efficient application of the technique while, in some cases, producing semantically more coherent image reconstructions. We apply our method to large-scale transformer-based vision models, specifically Detection Transformer, Vision Transformer, Swin Transformer, and Data-Efficient Image Transformer, analyzing the resulting reconstructions across network depth. Our main analysis compares Detection Transformer and Vision Transformer, which exhibit the most informative differences among the evaluated architectures. At the same time, results for Swin Transformer and Data-Efficient Image Transformer support the broader applicability of our framework. Our findings reveal gradual representational changes across transformer layers as a shared characteristic of Detection Transformer and Vision Transformer, as well as systematic differences in how they preserve contextual shape and fine-grained image details, and in their robustness to color perturbations. These findings deepen understanding of transformer-based vision models and demonstrate the utility of modular feature inversion as an interpretability tool.
| Comments: | Published in Transactions on Machine Learning Research (TMLR), September 2026, available at: this https URL . Code available at : this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE) |
| Cite as: | arXiv:2412.06534 [cs.CV] |
| (or arXiv:2412.06534v5 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2412.06534 arXiv-issued DOI via DataCite |
|
| Journal reference: | Transactions on Machine Learning Research (TMLR), 2026, ISSN 2835-8856 |
Submission history
From: Shirin Reyhanian [view email]
[v1]
Mon, 9 Dec 2024 14:43:06 UTC (15,874 KB)
[v2]
Fri, 21 Mar 2025 11:50:06 UTC (44,654 KB)
[v3]
Tue, 25 Mar 2025 10:19:48 UTC (44,654 KB)
[v4]
Wed, 13 Aug 2025 18:10:14 UTC (46,234 KB)
[v5]
Fri, 9 Oct 2026 11:02:34 UTC (42,292 KB)
来源：arXiv cs.AI · arxiv.org