arXiv cs.AI· Anisha Saha, Varsha Suresh, Teodora Kamova, Sophia Wiedmann, Timothy Hospedales, Vera Demberg·· 12 小时前AI 评分31
MuPHI 基准揭示视觉语言模型组合式有害性推理缺陷，CHIL 方法提升 OOD 泛化
Learning to Reason About Implicit Multimodal Harm: From Diagnosis to Generalization
AI 导读
研究者提出 MuPHI 诊断基准，专门考察图像与文本单独无害、组合后才产生有害性的跨模态推理，发现视觉语言模型能识别各模态证据，却无法将其绑定为有害机制。基于此提出的 CHIL 方法不再只优化预测标签，而是奖励证据识别、跨模态绑定与判定一致性，在 81% 的 OOD 迁移评测上平均提升 +2.5 macro-F1，反事实翻转率最高提升 +3.8 和 +5.4 个百分点。
正文 · 原文
Abstract:A benign image paired with a benign caption can become harmful only in combination, making detection fundamentally a cross-modal reasoning problem. Existing multimodal harm benchmarks evaluate only classification accuracy, leaving unclear whether models identify the cross-modal mechanism that produces harm. We introduce MuPHI (Multimodal Pragmatic Harm Interpretation), a controlled diagnostic benchmark in which harm arises purely from image-text composition, with benign counterfactuals and annotated harm rationales. Using MuPHI, we systematically evaluate where compositional harm reasoning breaks in vision-language models (VLMs). MuPHI reveals that VLMs reliably ground each modality yet fail to bind the evidence into the harm mechanism and their predictions respond weakly to label-determining counterfactual interventions. Moreover, existing harm detection approaches trained on labels substantially improves in-domain (ID) detection but transfers poorly across harm distributions. We therefore propose Compositional Harm Interpretation Learning (CHIL), which rewards evidence identification in each modality, cross-modal binding and verdict consistency in the reasoning chain rather than just predicted label. CHIL improves 81% OOD transfer evaluations by an average of +2.5 macro-F1 and correct counterfactual flip rate by up to +3.8 and +5.4 % points, while also improving overall reasoning quality over label-only optimization. These results suggest that optimizing how a model reasons, not just what it concludes, offers a promising direction for multimodal safety systems that generalize beyond dataset-specific correlations.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG); Multimedia (cs.MM) |
| Cite as: | arXiv:2605.29951 [cs.AI] |
| (or arXiv:2605.29951v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.29951 arXiv-issued DOI via DataCite |
Submission history
From: Anisha Saha [view email]
[v1]
Thu, 28 May 2026 13:58:15 UTC (5,428 KB)
[v2]
Fri, 16 Oct 2026 16:01:16 UTC (5,903 KB)
来源：arXiv cs.AI · arxiv.org