跳到正文
arXiv cs.CL· Jongbhin Woo, Youngjoon Jang, Joon Son Chung·· 9 小时前AI 评分18

SignAligner:用双粒度片段生成实现手语对齐

SignAligner: Dual-Granularity Span Generation for Sign Language Alignment

AI 导读

SignAligner 是一个提示词条件化的视觉-语言片段生成器,可将手语视频与字幕句子或目标词查询映射为文本形式的起止帧区间,先联合训练字幕片段与手语定位的词片段,再用重叠度奖励精调。在 BOBSL 字幕测试集上,精调后模型达到 66.68 F1@0.50,为无外部手语视频资源对比系统中最优。联合训练词片段可提升字幕对齐效果,且该优势在重叠度奖励精调后仍保留。

正文 · 原文

View PDF HTML (experimental)

Abstract:Large sign-language corpora benefit from temporal labels at two scales: subtitle spans for subtitle alignment and word spans for sign spotting. Sign spottings already locate words in time at corpus scale, but training subtitle alignment jointly on subtitle and word spans remains underexplored. We present SignAligner, a prompt-conditioned visual-language span generator that maps a sign-language video and a subtitle-sentence or target-word query to a textual start/end frame interval. It is first trained jointly on subtitle spans and word spans from sign spottings, then refined with overlap-based rewards while retaining auxiliary word examples. On the subtitle test set of the British Broadcasting Corporation-Oxford British Sign Language (BOBSL) corpus, the refined model reaches 66.68 F1@0.50 (F1 with a temporal intersection-over-union threshold of one half), the best among compared systems without external sign-video resources. This overlap-reward refinement outperforms cross-entropy continuation from the same jointly trained model. Joint training with word spans improves subtitle alignment in a pair of our generators matched in code, labels and maximum update budget. The advantage remains after both models of the pair are refined with overlap rewards. For word queries, the refined model follows the queried word, not a fixed position in the video: when the word's sign lies away from the video centre, it is far more accurate than predictions that use position alone. Together, these findings support one shared span generator refined with overlap rewards, aligning subtitles and producing word spans for human review.
Comments: 11 pages, 5 figures
Subjects: Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)
Cite as: arXiv:2611.22409 [cs.CV]
  (or arXiv:2611.22409v1 [cs.CV] for this version)
  https://doi.org/10.48550/arXiv.2611.22409

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Jongbhin Woo [view email]
[v1] Tue, 17 Nov 2026 08:47:09 UTC (1,503 KB)

来源:arXiv cs.CL · arxiv.org