arXiv cs.AI· Anum Afzal, Yuki Saito, Hiroya Takamura, Katsuhito Sudoh, Shinnosuke Takamichi, Graham Neubig, Florian Matthes, Tatsuya Ishigaki·· 10 天前AI 评分27
多模态 LLM 实时游戏视频解说：基于停顿感知的解码方法
Real-Time Generation of Game Video Commentary with Multimodal LLMs: Pause-Aware Decoding Approaches
AI 导读
研究提出两种基于提示词的解码策略实现多模态 LLM 实时游戏视频解说：固定间隔解码与动态间隔解码，后者根据上一句话的预估时长调整下一次预测时机，无需微调即可实现停顿感知生成。在日语和英语赛车、格斗游戏数据集上，动态间隔解码生成的解说在时间与内容上更贴近人类解说。团队同时发布了多语言基准数据集、训练模型与实现代码。
正文 · 原文
Abstract:Real-time video commentary generation provides textual descriptions of ongoing events in videos. It supports accessibility and engagement in domains such as sports, esports, and livestreaming. Commentary generation involves two essential decisions: what to say and when to say it. While recent prompting-based approaches using multimodal large language models (MLLMs) have shown strong performance in content generation, they largely ignore the timing aspect. We investigate whether in-context prompting alone can support real-time commentary generation that is both semantically relevant and well-timed. We propose two prompting-based decoding strategies: 1) a fixed-interval approach, and 2) a novel dynamic interval-based decoding approach that adjusts the next prediction timing based on the estimated duration of the previous utterance. Both methods enable pause-aware generation without any fine-tuning. Experiments on Japanese and English datasets of racing and fighting games show that the dynamic interval-based decoding can generate commentary more closely aligned with human utterance timing and content using prompting alone. We release a multilingual benchmark dataset, trained models, and implementations to support future research on real-time video commentary generation.
| Comments: | Accepted at LREC2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.02655 [cs.CL] |
| (or arXiv:2603.02655v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2603.02655 arXiv-issued DOI via DataCite |
Submission history
From: Tatsuya Ishigaki [view email]
[v1]
Tue, 3 Mar 2026 06:39:04 UTC (6,051 KB)
[v2]
Tue, 6 Oct 2026 13:18:40 UTC (11,506 KB)
来源：arXiv cs.AI · arxiv.org