arXiv cs.AI· Lubnaa Abdur Rahman, Ioannis Papathanail, Maria Panagiotou, Nefeli Dimitropoulou, Margret Har{\dh}ard\'ottir, Elisabeth J. den Brok, Bastiaan E. de Galan, Stavroula Georgia Mougiakakou·· 12 小时前AI 评分31
NutriVisRAG：用视觉相似餐食的多模态示例为 VLM 营养估算提供依据
NutriVisRAG: Multimodal Exemplar Grounding for Nutrient Estimation with Vision-Language Models
AI 导读
NutriVisRAG 是一个检索增强生成框架，用视觉编码器检索带营养标注的相似餐食，作为多模态上下文示例供 VLM 在推理时估算营养，无需任务特定微调。在 4 个数据集、4 个 VLM、3 个检索编码器和 2 种提示策略共 96 种配置中，56 种配置的主误差下降，整餐提示下为 35/48，逐项提示下为 21/48。
正文 · 原文
Abstract:Diet is a major determinant of health, yet accurate dietary assessment from free-living meal images remains challenging. Existing retrieval-based Vision-Language Model (VLM) approaches to nutrient estimation typically rely on textual food representations, while, to our knowledge, direct grounding on visually similar annotated meals has not been investigated. We present NutriVisRAG, a retrieval-augmented generation framework that uses a vision encoder to retrieve similar meals with known nutritional annotations and provide them as multimodal in-context exemplars for VLM nutrient estimation at inference time, without task-specific fine-tuning. We evaluate NutriVisRAG across four datasets, four VLMs, three retrieval encoders, and two prompting strategies, including free-living meals from adults with insulin-treated diabetes, where carbohydrate estimates are compared against two dietitians' annotations. Across all 96 configurations (4 datasets x 4 VLMs x 3 retrieval encoders x 2 prompting strategies), NutriVisRAG reduced the primary estimation error in 56, with improvements more frequent under whole-meal (35/48) than per-item prompting (21/48). Across the evaluated configurations, the lowest NutriVisRAG error was below the best Vanilla error in 13 of 16 VLM--dataset pairs. For GPT-5.4-mini, under whole-meal prompting, the best-performing retrieval encoder for each dataset reduced the primary error on all four datasets relative to Vanilla, with reductions of 1.7--35.7%. Overall, our results show that grounding in multimodal exemplars comprising images and respective annotations can improve nutrient estimation at inference time without task-specific fine-tuning, although the benefits remain configuration-dependent.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.08054 [cs.CV] |
| (or arXiv:2611.08054v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.08054 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lubnaa Abdur Rahman [view email]
[v1]
Thu, 5 Nov 2026 14:10:35 UTC (2,421 KB)
来源：arXiv cs.AI · arxiv.org