arXiv cs.CL· Junyi Hu, Zhewen He, Haomian Huang, Zhenhua Li, Zhifei Li, Shuaihang Yuan, Yi Fang·· 12 小时前AI 评分31
SignDEG：用 DINO 自监督学习吸收手部关节几何的手语表征
DINO Eats Geometry: Self-Supervised Sign Representations that Absorb Articulated Hand Geometry
AI 导读
SignDEG 是一种时序手语编码器，以手部关节几何为预训练目标，推理时仅需视频。在 SHuBERT 公开数据评测协议下，How2Sign/OpenASL 上 BLEU-4 达 17.9/24.5，高于 SHuBERT 的 16.2/23.2；Rank-1 适配使 ASL Citizen 与 Sem-Lex 的 Recall@1 各提升 0.02。
正文 · 原文
Abstract:Handshape and orientation are geometric components of sign language, yet many sign encoders learn from appearance alone. We introduce SignDEG (DINO Eats Geometry), a temporal sign encoder that uses articulated hand geometry as a pre-training target and requires only video at inference. Frozen DINOv3 features of tracked hands and faces feed per-stream temporal Transformers. Geometric latent prediction (GLP) supervises an intermediate block with WiLoR latents, MANO articulation, and 3D joints, including at masked frames. Geometric relation transfer (GRT) matches cross-video frame-similarity distributions at the final block. Both objectives are learned jointly with temporal self-distillation; WiLoR and the geometry heads are then discarded. Under the SHuBERT public-data evaluation protocol, SignDEG reaches 17.9/24.5 BLEU-4 on How2Sign/OpenASL, compared with 16.2/23.2 for SHuBERT. Rank-1 adaptation improves ASL Citizen and Sem-Lex recognition by 0.02 Recall@1 each, and phonological classification improves across 16 attributes. Removing GRT while retaining GLP reduces How2Sign BLEU-4 by 1.4 and ASL Citizen Recall@1 by 0.034. The results support transferring articulated structure into an appearance encoder without running a hand reconstruction model at deployment.
| Comments: | 22 pages, 7 figures |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL) |
| Cite as: | arXiv:2609.06296 [cs.CV] |
| (or arXiv:2609.06296v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2609.06296 arXiv-issued DOI via DataCite |
Submission history
From: Junyi Hu [view email]
[v1]
Sat, 5 Sep 2026 23:08:37 UTC (5,095 KB)
[v2]
Thu, 15 Oct 2026 20:29:41 UTC (3,039 KB)
来源：arXiv cs.CL · arxiv.org