arXiv cs.CL· Kris Kraack·· 2 天前AI 评分22
多模态情感识别系统：融合面部表情、肢体动作、语音与口语文本
A Multimodal Emotion Recognition System: Integrating Facial Expressions, Body Movement, Speech, and Spoken Language
AI 导读
一个融合面部表情、语音、转录文本与肢体动作的多模态情感识别系统在52名参与者的远程用户研究中达到96.43%的准确率，显著高于任一单模态组件（p < 10⁻⁴），且各情绪状态识别率均不低于92.31%。相比之下，单模态模型在真实条件下准确率降至82.97%–87.64%，部分情绪状态识别率低至48.08%，表明基准测试的高性能难以迁移到实际使用场景。
正文 · 原文
Abstract:Psychological evaluation relies on clinicians' observation and interpretation of a patient's emotional expression, which is susceptible to subjectivity, bias, fatigue, and inconsistency. This paper presents a multimodal emotion recognition system designed to support, rather than replace, the evaluator. The system combines facial expression recognition, speech emotion recognition, text-based emotion recognition of transcribed speech, and body movement analysis, and fuses their outputs into an emotional profile of class probabilities that is presented to the evaluator together with the overall prediction. Although the unimodal models achieve high accuracy on their benchmark test sets, a remote user study with 52 participants expressing seven emotional states shows that their accuracy falls to between 82.97% and 87.64% under realistic conditions, with individual emotional states recognised in as few as 48.08% of cases. Under the same conditions, the multimodal system achieves an accuracy of 96.43%, significantly higher than every single-modality component ($p < 10^{-4}$), with no emotional state below 92.31%. These results indicate that strong benchmark performance of a single channel does not transfer to realistic use, and that multimodal fusion provides the robustness required of a tool that supports psychological evaluation.
| Comments: | v2: Models revised to correct issues in the original implementation and results updated accordingly; text revised and restructured, architecture figures added, fusion stage formalised, references updated. 8 pages, 4 figures, 8 tables |
| Subjects: | Human-Computer Interaction (cs.HC); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Multimedia (cs.MM); Sound (cs.SD); Audio and Speech Processing (eess.AS) |
| Cite as: | arXiv:2412.17907 [cs.HC] |
| (or arXiv:2412.17907v2 [cs.HC] for this version) | |
| https://doi.org/10.48550/arXiv.2412.17907 arXiv-issued DOI via DataCite |
Submission history
From: Kris Kraack [view email]
[v1]
Mon, 23 Dec 2024 19:00:34 UTC (505 KB)
[v2]
Thu, 15 Oct 2026 14:23:34 UTC (20 KB)
来源：arXiv cs.CL · arxiv.org