arXiv cs.AI· Ziniu Zhang, Zhenshuo Zhang, Jianglin Lu, Ruoxuan Xiong, Yun Fu, Hongyang R. Zhang·· 10 小时前AI 评分24
结构化核代理模型 SKS 实现跨样本多模态归因
Structured Kernel Surrogate Models for Multimodal Attribution
AI 导读
研究者提出结构化核代理模型与 SKS 算法，用结构化组合核在样本与模态的选择状态上做核岭回归，衡量一个样本的某模态如何影响其他多模态样本的学习结果。在两个多模态基准上，SKS 与真实预测结果的相关性比现有方法高 57.8%，下游数据选择平均准确率提升 1.93%。用 SKS 筛选卫星图像时，模型会挑出事故常发路段而非背景图像，且所选数据训练的模型 Hessian 迹更低、损失面更平坦。
正文 · 原文
Abstract:We study the problem of cross-sample modality attribution, which involves measuring how one modality in one sample affects the learning outcome in other multimodal samples. This problem is crucial in a multimodal system because a data modality (such as a text description) may be redundant for one sample (a self-explanatory image), but the same data modality would otherwise be uniquely informative for another sample, or informative when combined with other modalities. Prior work focuses on influence estimation at the level of one sample, and it remains an open question how one should capture the interactions across both samples and modalities. To capture scenarios like the above, we propose structured kernel surrogate models, which are surrogate mappings from both samples and modalities to the prediction outcome when they are trained together. We design an algorithm, SKS, which applies kernel ridge regression with a structured, compositional kernel on the selection states of individual samples and modalities. Specifically, the kernel function multiplies kernel-based similarity scores between every pair of data modalities for one sample and a pair of two samples. Additionally, we design a gradient-based estimation algorithm to efficiently estimate the structured kernel surrogate model. We evaluate our approach on two multimodal benchmarks, spanning a range of data modalities including satellite images, road networks, text, audio, and tabular data. We find that SKS achieves $57.8$% higher correlation with the true prediction outcomes than existing methods. In downstream data selection, SKS improves the average accuracy by $1.93$% over baselines. Models with data selected by SKS have lower Hessian traces, indicating a flatter loss surface. Using SKS to select satellite images identifies road segments, where accident scenes commonly occur, as opposed to background images.
| Comments: | 20 pages. To appear at NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2611.13198 [cs.LG] |
| (or arXiv:2611.13198v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13198 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ziniu Zhang [view email]
[v1]
Mon, 9 Nov 2026 20:21:39 UTC (3,771 KB)
来源：arXiv cs.AI · arxiv.org