arXiv cs.AI· Dan Ben-Ami, Gabriele Serussi, Kobi Cohen, Chaim Baskin·· 14 小时前AI 评分34
HiMu：面向长视频问答的层次化多模态帧选择框架
HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering
AI 导读
HiMu 是一个免训练的多模态帧选择框架，用单次纯文本 LLM 调用将查询分解为层次逻辑树，叶子节点路由到视觉（CLIP、开放词汇检测、OCR）与音频（语音识别、非语音声音匹配）专家，再经模糊逻辑算子自底向上组合出逐帧满足度曲线。
正文 · 原文
Abstract:Long-form video question answering requires reasoning over extended temporal contexts, making frame selection a critical bottleneck for multi-modal large language models (MLLMs) bound by finite context windows. Within the controlled frame-budget regime that governs practical deployment, prior selectors score frames against a single global query embedding; as a result, compositional multimodal questions that involve temporal ordering or cross-modal cues such as "what happens on screen right after the narrator mentions the reaction?" are flattened into a representation that loses sub-event ordering and modality bindings. We introduce HiMu, a training-free framework for compositional multimodal frame selection. A single text-only LLM call decomposes the query into a hierarchical logic tree whose leaves are atomic predicates, each routed to a lightweight expert spanning vision (CLIP, open-vocabulary detection, OCR) and audio (speech recognition and non-speech sound matching). Expert signals are normalized, smoothed to align across modalities, and composed bottom-up through fuzzy-logic operators that enforce temporal sequencing and adjacency, yielding a continuous per-frame satisfaction curve. Under the standard 16-frame budget on Video-MME, LongVideoBench, and HERBench-Lite, HiMu achieves state-of-the-art accuracy among frame selection methods and improves over uniform sampling across seven diverse MLLMs as a drop-in module, matching the accuracy of uniform sampling at 4x its frame budget, without retraining and without multiple iterative MLLM calls during selection.
| Comments: | Accepted to WACV 2027. Project page: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.18558 [cs.CV] |
| (or arXiv:2603.18558v3 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2603.18558 arXiv-issued DOI via DataCite |
Submission history
From: Dan Ben Ami [view email]
[v1]
Thu, 19 Mar 2026 07:11:53 UTC (26,841 KB)
[v2]
Fri, 26 Jun 2026 08:15:42 UTC (29,262 KB)
[v3]
Sun, 25 Oct 2026 06:01:15 UTC (6,517 KB)
来源：arXiv cs.AI · arxiv.org