arXiv cs.CL· Ankit Gupta, Akshat Pandya, Christoph Adami, Emily Dolson·· 12 小时前AI 评分31
SPOOK：视觉编码器离流形知识的稀疏像素优化
SPOOK: Sparse Pixel Optimization for Off-Manifold Knowledge in Vision Encoders
AI 导读
研究者提出 SPOOK，用像素级爬山法对空白画布做稀疏像素编辑，在 ImageNet 和 Flickr30k 上测试多模态对比编码器与自监督视觉编码器能否检索到指定目标。
正文 · 原文
Abstract:Modern vision encoders are widely used as semantic representations, but the relationship between embedding similarity and human-visible semantics can become unclear far from the natural-image manifold. We probe this regime in a black-box setting by applying sparse pixel edits to blank canvases with a simple pixel-level hill-climbing procedure. Across ImageNet and Flickr30k, we evaluate whether these edited blank canvases can retrieve chosen targets in multimodal contrastive encoders and self-supervised visual encoders. On OpenCLIP, inputs optimized toward class prompts achieve strong top-1 and top-5 targeted retrieval performance; on DINOv3, inputs optimized toward class prototypes achieve high target alignment and near-top prototype ranks. These results show that sparse edits on otherwise blank, visually non-semantic inputs can enter target semantic basins in both language-aligned and self-supervised visual representation spaces. Our findings identify a shared off-manifold failure mode of modern vision encoders and suggest sparse blank-canvas optimization as a simple probe of representation geometry.
| Comments: | 12 pages, 3 figures, 3 tables, and 14 pages of Supplementary Material with 8 figures |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL); Human-Computer Interaction (cs.HC) |
| Cite as: | arXiv:2610.32835 [cs.CV] |
| (or arXiv:2610.32835v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32835 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Christoph Adami [view email]
[v1]
Fri, 23 Oct 2026 16:09:57 UTC (28,849 KB)
来源：arXiv cs.CL · arxiv.org