arXiv cs.AI· Subhashini Venugopalan, Jenny Tan, Taylor Roper, Jimmy Tobin, Anton Kast, Alicia Martin, Sam Sepah, Amy Pavel·· 14 小时前AI 评分31
MTG-AD：面向端到端内联音频描述的细粒度评测数据集
Mind The Gap (MTG-AD): Dataset and Fine-grained Evaluation for End-to-End Inline Audio Descriptions
AI 导读
研究者发布 MTG-AD，一个由盲人与视力正常音频描述专家共同标注的 36 小时视频数据集，含 8k+ 条描述，用于评测 MLLM 生成的内联音频描述。配套的 MTG-AD-Eval 框架将生成脚本渲染为音轨，评估其在视频自然静音段内的贴合度。测试显示前沿 MLLM 能准确描述视觉事件并大幅减少语音重叠，但在叙事节奏与流畅度上仍有差距。
正文 · 原文
Abstract:Audio descriptions (AD) are essential for making visual media accessible to blind and low-vision (BLV) users. While Multimodal Large Language Models (MLLMs) offer a scalable solution for generating audio descriptions on-demand, their performance on ``inline'' audio descriptions -- which must fit within existing silences in a video -- remains under-explored, particularly for diverse user-generated content (UGC). We present a comprehensive investigation into MLLM-generated inline audio descriptions. First, we introduce MTG-AD, a professionally annotated dataset of 36 hours of videos, each averaging $\sim$5 mins. and 8k+ descriptions in total, co-created by paired blind and sighted AD professionals to provide a high-quality benchmark for this task. Second, we propose MTG-AD-Eval, an evaluation framework based on audio description expert guidelines that provides fine-grained, reference-free actionable metrics for model improvement. Crucially, our framework evaluates the end-to-end task: it renders the generated scripts into audio tracks to assess their fit within the original video's natural silences, minimizing disruptive overlaps. Our analysis across frontier MLLMs reveals that while models accurately describe visual events and substantially reduce speech overlaps, gaps persist in narrative pacing and flow. Finally, we empirically validate that our automated and reference-free metrics correlate significantly with human quality ratings from sighted raters and 7 BLV participants, finding that while users narrowly prefer human-authored audio descriptions, they find MLLM-generated audio descriptions highly beneficial for comprehension. We provide actionable system design guidance and release MTG-AD at this https URL and MTG-AD-Eval at this https URL.
| Comments: | NeurIPS 2026 Evaluations & Datasets Track, camera ready |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC) |
| Cite as: | arXiv:2610.39955 [cs.CV] |
| (or arXiv:2610.39955v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39955 arXiv-issued DOI via DataCite |
|
| Journal reference: | Advances in Neural Information Processing Systems 2026 |
Submission history
From: Subhashini Venugopalan [view email]
[v1]
Thu, 29 Oct 2026 12:51:38 UTC (4,641 KB)
来源：arXiv cs.AI · arxiv.org