arXiv cs.AI· Huu Binh Ta, Alex Morehead, N. Benjamin Erichson, Ferdinando Fioretto, Sandeep Madireddy·· 10 小时前AI 评分20
TCAF:面向多模态材料语言模型的任务感知表征融合
Choosing the Right Representations: Task-Aware Fusion for Multimodal Materials Language Models
AI 导读
研究提出 Task-Conditioned Attention Fusion(TCAF),一种轻量注意力桥接模块,用任务条件化的隐查询将 CHGNet 特征与 Zatom-1 中间层表征结合,直接在下游预测任务上训练,无需单独的对比预训练。
正文 · 原文
Abstract:Multimodal language models for materials science rely on effective mechanisms for incorporating three-dimensional structural representations. MatterChat addresses this challenge using a large bridge module between a pretrained structural encoder and the LLM, together with a separate contrastive alignment stage. We instead ask whether this bridge can adaptively select and combine representations from multiple structural encoders according to the target property. We find that supervised and generative material encoders provide complementary information for property prediction. We found that, for Zatom-1, this information is not concentrated in the final layer. Instead, the most informative representations occur at different depths depending on the target property. This motivates Task-Conditioned Attention Fusion (TCAF), a lightweight attention-based bridge that uses task-conditioned latent queries to combine CHGNet features with intermediate Zatom-1 representations. The bridge is trained directly on downstream prediction tasks, allowing representation selection and material-language alignment to be learned jointly without separate contrastive pretraining. Across classification and regression tasks, TCAF outperforms MatterChat, with up to 47.8% lower RMSE, while using substantially fewer trainable parameters and halving the overall training time. Further experiments incorporating UMA-S demonstrate TCAF's flexibility across encoder combinations, with the three-encoder configuration achieving the highest macro-average classification accuracy of 89.61%.
| Comments: | Accepted at the AI4Mat Workshop at NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.24641 [cs.LG] |
| (or arXiv:2611.24641v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24641 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Huu Binh Ta [view email]
[v1]
Wed, 18 Nov 2026 15:48:25 UTC (262 KB)
来源:arXiv cs.AI · arxiv.org