arXiv cs.CL· Juyi Lin, Zhiqiang Lao, Jiali Cui, Lin Zhao, Pu Zhao, Dichang Zhang, Arman Akbari, Yu Qi, Xinru Jiang, Yanzhi Wang, Heather Yu, Liang Peng·· 2 天前AI 评分22
LEAP：面向长音视频感知的学习式分块证据检索框架
LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perception
AI 导读
LEAP 通过将录音切分为固定时长块并先做轻量定位、再对高分窗口做有界重编码答题，使答题输入与峰值上下文不随录制时长增长。在多个 AVQA 基准上，LEAP 较 Qwen3-Omni-30B-A3B 基线提升 4.5-16.8%，并在 MiniCPM-o 4.5 上超出其已公布结果 3.1-13.0%。
正文 · 原文
Authors:Juyi Lin, Zhiqiang Lao, Jiali Cui, Lin Zhao, Pu Zhao, Dichang Zhang, Arman Akbari, Yu Qi, Xinru Jiang, Yanzhi Wang, Heather Yu, Liang Peng
Abstract:Hour-scale audio-visual question answering is constrained by a context dilemma: dense whole-recording encoding rapidly exhausts context limits, whereas uniform temporal compression severely dilutes fine-grained acoustic and visual evidence. We introduce LEAP, a framework where the model retrieves its own evidence without placing the whole recording in one context. LEAP divides a recording into fixed-duration blocks, applying a lightweight localization pass to each block to score short candidate windows. The highest-ranked windows are pooled and re-encoded in a single bounded answer pass. Consequently, the answer input and peak context remain independent of the recording duration. By decoupling evidence localization from reasoning, our framework can localize candidate temporal windows over pre-computed transcripts without decoding media frames, while preserving fine-grained visual and non-speech evidence by routing the final answering pass over raw audio-visual streams. LEAP trains both stages: a localization LoRA improves the selected windows, and an answer LoRA improves the answers read from the same windows. The block grid natively supports causal queries, enabling LEAP to support streaming inference without streaming-specific training. Across several AVQA benchmarks, LEAP improves over the Qwen3-Omni-30B-A3B baseline by 4.5-16.8%, and transfers to a second omni-modal backbone, MiniCPM-o 4.5, surpassing its published results by 3.1-13.0%.
| Comments: | 39 pages, 16 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2609.39938 [cs.CL] |
| (or arXiv:2609.39938v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.39938 arXiv-issued DOI via DataCite |
Submission history
From: Juyi Lin [view email]
[v1]
Wed, 30 Sep 2026 15:19:12 UTC (270 KB)
[v2]
Wed, 14 Oct 2026 23:53:17 UTC (271 KB)
来源：arXiv cs.CL · arxiv.org