arXiv cs.CL· Daniela Jimenez-Pano, Hugo Jair Escalante, Delia Iraz\'u Hern\'andez-Far\'ias·· 12 小时前AI 评分22
基于门控多模态表示的 RAG 医疗视觉问答增强方法
Enhancing RAG-based Medical VQA with Gated Multimodal Representations
AI 导读
研究者提出一种多模态 RAG 框架，通过 Gated Multimodal Unit 学习联合查询表示，让证据检索同时利用医学图像与临床问题的语义，且保持底层 Med-LVLM 冻结。在基准数据集上，该多模态查询表示持续优于参考 Med-LVLM，并与 SOTA 方法保持竞争力。
正文 · 原文
Abstract:Medical Vision-Language Models (Med-LVLMs) have shown strong performance in Medical Visual Question Answering (MVQA), yet they may still produce factually inconsistent responses. Retrieval-Augmented Generation (RAG) can mitigate this issue by incorporating external clinical evidence; however, retrieval is often guided by a single modality, potentially overlooking the joint semantics of medical images and clinical questions. We introduce a multimodal RAG framework that learns a joint query representation through a Gated Multimodal Unit, enabling evidence retrieval to leverage information from both modalities while keeping the underlying Med-LVLM frozen. We evaluate the proposed approach on benchmark datasets. Results show that the multimodal query representation consistently outperforms a reference Med-LVLM and remains competitive with state-of-the-art methods. Our findings demonstrate the potential of multimodal query representations for more effective evidence retrieval in Medical VQA.
| Comments: | Preprint submitted to ECIR |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.23588 [cs.CV] |
| (or arXiv:2610.23588v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.23588 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hugo Jair Escalante [view email]
[v1]
Fri, 16 Oct 2026 15:07:33 UTC (3,750 KB)
来源：arXiv cs.CL · arxiv.org