跳到正文
arXiv cs.AI· Huu Binh Ta, Alex Morehead, N. Benjamin Erichson, Ferdinando Fioretto, Sandeep Madireddy·· 10 小时前AI 评分20

TCAF:面向多模态材料语言模型的任务感知表征融合

Choosing the Right Representations: Task-Aware Fusion for Multimodal Materials Language Models

AI 导读

研究提出 Task-Conditioned Attention Fusion(TCAF),一种轻量注意力桥接模块,用任务条件化的隐查询将 CHGNet 特征与 Zatom-1 中间层表征结合,直接在下游预测任务上训练,无需单独的对比预训练。

正文 · 原文

View PDF HTML (experimental)

Abstract:Multimodal language models for materials science rely on effective mechanisms for incorporating three-dimensional structural representations. MatterChat addresses this challenge using a large bridge module between a pretrained structural encoder and the LLM, together with a separate contrastive alignment stage. We instead ask whether this bridge can adaptively select and combine representations from multiple structural encoders according to the target property. We find that supervised and generative material encoders provide complementary information for property prediction. We found that, for Zatom-1, this information is not concentrated in the final layer. Instead, the most informative representations occur at different depths depending on the target property. This motivates Task-Conditioned Attention Fusion (TCAF), a lightweight attention-based bridge that uses task-conditioned latent queries to combine CHGNet features with intermediate Zatom-1 representations. The bridge is trained directly on downstream prediction tasks, allowing representation selection and material-language alignment to be learned jointly without separate contrastive pretraining. Across classification and regression tasks, TCAF outperforms MatterChat, with up to 47.8% lower RMSE, while using substantially fewer trainable parameters and halving the overall training time. Further experiments incorporating UMA-S demonstrate TCAF's flexibility across encoder combinations, with the three-encoder configuration achieving the highest macro-average classification accuracy of 89.61%.
Comments: Accepted at the AI4Mat Workshop at NeurIPS 2026
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.24641 [cs.LG]
  (or arXiv:2611.24641v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2611.24641

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Huu Binh Ta [view email]
[v1] Wed, 18 Nov 2026 15:48:25 UTC (262 KB)

来源:arXiv cs.AI · arxiv.org