arXiv cs.AI· Corentin Caris, Louis Jalouzot, Nona Rajabi, G\'eraud Ilinca, Ahmed Hassayoune, Asilbek Muminov, Thadd\'ee Delebarre, Clara Fonteneau, Charles Maussion, Bertrand Thirion, Alexis Thual, Antoine Collas·· 2 天前AI 评分27
NOEMA：用时间建模与多模态对齐从 fMRI 解码自然语音
Decoding Naturalistic Speech from fMRI with Temporal Modeling and Multimodal Alignment
AI 导读
NOEMA 是一种时间卷积解码器，将连续 fMRI 测量窗口与同一刺激片段的文本、音频、语音多模态嵌入对齐，实现直接从脑活动检索语音片段。在 Cephalonauts One 和 LeBel 两个自然语音数据集上，超过 42% 的留出查询能在 2000 个候选中将正确片段检索进前 10，显著优于基线解码器，且候选池加入数千个训练中未出现的干扰项后优势依然保持。
正文 · 原文
Authors:Corentin Caris, Louis Jalouzot, Nona Rajabi, Géraud Ilinca, Ahmed Hassayoune, Asilbek Muminov, Thaddée Delebarre, Clara Fonteneau, Charles Maussion, Bertrand Thirion, Alexis Thual, Antoine Collas
Abstract:Reconstructing what a person perceives from their brain activity is a stringent test of whether neural recordings carry meaningful information. Naturalistic speech offers a rich testbed for this challenge: it unfolds over multiple timescales, but fMRI's noisy, temporally smeared measurements make this structure hard to recover. We introduce NOEMA, a temporal convolutional decoder that aligns windows of consecutive fMRI measurements with multimodal stimulus embeddings. These embeddings combine complementary text, audio, and speech representations of the same stimulus segment, enabling its retrieval directly from brain activity. On two naturalistic speech datasets, Cephalonauts One and LeBel, NOEMA retrieves the correct segment among 2,000 candidates within the top 10 for more than 42% of held-out queries. This substantially outperforms baseline decoders, an advantage that persists when the candidate pool is extended with thousands of distractors absent from training. Controlled ablations establish the contributions of temporal modeling and complementary stimulus representations.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.22049 [cs.LG] |
| (or arXiv:2610.22049v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22049 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Antoine Collas [view email]
[v1]
Thu, 15 Oct 2026 15:17:15 UTC (1,435 KB)
来源：arXiv cs.AI · arxiv.org