Semantic Motion Anchors：用语义动作锚点连接语音手势的动作与含义

View PDF HTML (experimental)

Abstract:Learning a shared representation between spoken text and gesture is central to co-speech gesture retrieval, synthesis, and understanding, but remains challenging for semantically meaningful gestures whose communicative intent is not captured by motion alone. Direct contrastive alignment between transcripts and continuous motion embeddings often overemphasizes low-level kinematics and misses the symbolic content of semantic gestures. We propose semantic motion anchors, natural-language abstractions of gesture motion capturing physical form and communicative intent. Our method discretizes 3D gestures into body-hand motion primitives, verbalizes them into structured descriptions, and grounds them in the transcript to provide auxiliary contrastive supervision. On BEAT2, our method improves text-to-gesture R@1 by 8.2% over a direct text-motion baseline and outperforms prior retrieval approaches on text to gesture and gesture to text retrieval directions. Beyond aggregate retrieval metrics, semantic motion anchor supervision helps retrieve gestures that are semantically meaningful for the spoken query, rather than defaulting to generic motion patterns. A downstream retrieval-augmented gesture generation study showed that users significantly preferred gestures retrieved by our approach over a retrieval-augmented generation baseline, demonstrating that semantically grounded retrieval translates to gestures that better convey communicative intent in downstream generation.

Comments:	NeuRIPS 2026
Subjects:	Computation and Language (cs.CL)
Cite as:	arXiv:2605.30608 [cs.CL]
	(or arXiv:2605.30608v4 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2605.30608 arXiv-issued DOI via DataCite

Submission history

From: Varsha Suresh [view email]
[v1] Thu, 28 May 2026 22:01:11 UTC (2,355 KB)
[v2] Mon, 1 Jun 2026 14:43:09 UTC (2,363 KB)
[v3] Sat, 6 Jun 2026 20:59:52 UTC (2,355 KB)
[v4] Mon, 9 Nov 2026 21:37:28 UTC (2,584 KB)