arXiv cs.AI· Jianpeng Cheng, Guangyu Sun, Aashu Singh, Benyu Zhang, Haixing Dai, Hossein Mansour, Jiangfan Zhang, Shlok Kumar Mishra, Wei Sun, Xuanming Cui, Yanli Liu, Qi Guo, Max Xiangjun Fan, Jun Xiao·· 9 天前AI 评分27
MetaEncoder：用自然语言界面探索双编码器在多模态 System One 决策中的极限
MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface
AI 导读
MetaEncoder 将预训练的 Muse-Glimmer 30B 解码器微调为指令遵循的决策编码器，采用双编码器架构和单向对比学习实现请求与候选选项的对齐。该方法同时支持小于 256 的闭集候选和百万级开集候选，在 11 个基准套件、190 项任务上评测，覆盖多模态决策、理解与检索，并在推理密集型任务上暴露出当前局限。
正文 · 原文
Authors:Jianpeng Cheng, Guangyu Sun, Aashu Singh, Benyu Zhang, Haixing Dai, Hossein Mansour, Jiangfan Zhang, Shlok Kumar Mishra, Wei Sun, Xuanming Cui, Yanli Liu, Qi Guo, Max Xiangjun Fan, Jun Xiao
Abstract:System One models output constrained decisions and probability distributions rather than free-form text generation. While prevailing paradigms rely on structured schema objects to encode state, intent, and candidate choices, we revisit a fully natural language-based System One interface. In this framework, both the user request and each candidate option are expressed in natural language, supported by multimodal (image and video) auxiliary inputs. We introduce MetaEncoder, which fine-tunes a pre-trained Muse-Glimmer 30B decoder into an instruction-following decision-making encoder. To scale effectively across both small closed-set (< 256) and massive open-set (millions) candidate spaces, MetaEncoder employs a bi-encoder architecture trained via unidirectional contrastive learning for request-candidate alignment. We conduct extensive evaluations across 11 benchmark suites and 190 tasks spanning multimodal decision-making, understanding (closed-set) and retrieval (open-set), highlighting where MetaEncoder beats SOTA multimodal encoders, as well as its current limits on reasoning-intensive tasks.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.11316 [cs.CL] |
| (or arXiv:2610.11316v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11316 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jianpeng Cheng J [view email]
[v1]
Thu, 8 Oct 2026 06:22:48 UTC (324 KB)
来源：arXiv cs.AI · arxiv.org