ConstraintMatch：通过约束区域证据选择实现组合式图文匹配

View PDF HTML (experimental)

Abstract:Local image--text matching can find strong evidence for individual concepts yet favor captions with incorrect attribute--entity bindings. We introduce ConstraintMatch, a training-free inference method that turns entity-level regional consistency into an explicit matching cost. On the same frozen condition--region similarity matrix, it compares independent crop selection for each category and attribute with a shared crop choice within each entity. Their gap quantifies the score advantage obtained by relaxing the shared-region constraint. Spatial references from the same image calibrate this gap by keeping category evidence fixed while jointly shifting an entity's attribute maps. Per-entity clipping prevents cancellation across residuals, and the scaled mean cost is subtracted from global similarity. ConstraintMatch uses pooled embeddings, and spatial calibration requires no additional encoder calls. With parameters fixed per backbone across tasks, ConstraintMatch improves over global matching by 7.70 percentage points on full ARO Attribute and 5.63 points on held-out SugarCrepe++ attribute swaps, averaged across Qwen~2B and CORE~2B. Against SGI adapted to the same 13-crop budget, the respective gains are 4.85 and 3.51 points. Controlled analyses distinguish the effects of shared-region matching and candidate score calibration. The results establish entity-level evidence selection as a practical scoring principle for compositional multimodal retrieval.

Subjects:	Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)
Cite as:	arXiv:2611.00784 [cs.CV]
	(or arXiv:2611.00784v1 [cs.CV] for this version)
	https://doi.org/10.48550/arXiv.2611.00784 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Yuhong Zhou [view email]
[v1] Sat, 31 Oct 2026 09:57:06 UTC (2,517 KB)