arXiv cs.AI· Avinash Anand, Aman Kumar, Chaitanya Lakhchaura, Ashutosh Kumar, Akshita Abrol, Timothy Liu, Zhengkui Wang·· 12 小时前AI 评分27
NutriVision：用食材条件融合与预测实现单图食物营养估算
NutriVision: Ingredient-Conditioned Fusion and Prediction for Single-Image Food Nutrition Estimation
AI 导读
NutriVision 是一个端到端框架，仅凭单张 RGB 图像和可选食材清单即可估算热量、质量、脂肪、碳水和蛋白质，并用 DepthAnything-V3 补出缺失的深度模态、用 CLIP 编码食材描述。
正文 · 原文
Abstract:Nutrition estimation is a fundamental task in consumer diet tracking, clinical dietetics, chronic disease management, sports and hospital nutrition, and broader food computing systems. The existing approaches have progressed along two largely separate axes, vision models that rely on calibrated RGB-depth captures and ingredient-aware methods that use textual cues but use limited multimodal fusion. We introduce NutriVision, an end-to-end framework that leverages visual geometry and ingredient semantics to estimate calories, mass, fat content, carbohydrates, and protein from a single RGB image and an optional ingredient list. It obtains the unavailable depth modality using DepthAnything-V3 and encodes ingredient descriptions using CLIP. It integrates three complementary mechanisms: (1) an \emph{Ingredient-Conditioned Frequency-Aligned Fusion Module (IC-FAFM)}, which uses textual guidance to reweight and align RGB-depth frequency components; (2) an \emph{Ingredient-Aware Mask-based Prediction Head (IA-MPH)}, whose gating and channel masks are conditioned on food identity; and (3) modality-specific \emph{Internal Semantic Modeling (ISM)} blocks. On the Nutrition5k dataset, NutriVision achieves a mean PMAE of $\mathbf{13.60\pm0.10\%}$, outperforming our IGSMNet implementation by $0.89$ percentage points and OmniFood8k by $2.90$ percentage points (both $p<0.001$). The module-level ablations identify the ingredient-aware prediction head as the primary architectural contributor, improving mean PMAE by $1.50\pm0.17$ percentage points ($p<0.001$). These results demonstrate that ingredient-conditioned prediction and frequency-aware RGB-depth fusion provide measurable gains for single-image nutrient estimation. More broadly, NutriVision offers a practical route toward nutrition-assessment systems that exploit geometric and semantic cues without requiring specialized depth-sensing hardware
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.33076 [cs.CV] |
| (or arXiv:2609.33076v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2609.33076 arXiv-issued DOI via DataCite |
Submission history
From: Ashutosh Kumar [view email]
[v1]
Sun, 27 Sep 2026 01:16:34 UTC (5,300 KB)
[v2]
Fri, 16 Oct 2026 14:33:03 UTC (5,052 KB)
来源：arXiv cs.AI · arxiv.org