arXiv cs.AI· Ruimeng Liu, Denan Liang, Ruofei Bai, Fuming Zhang, Kuan Xu, Shenghai Yuan, Lihua Xie·· 10 小时前AI 评分34
PhysMoE：模态解耦 VLM 中用单 Token 坐标表示实现快速空间定位
PhysMoE: Fast Spatial Grounding with Single-Token Coordinate Representation in Modality-Decoupled VLMs
AI 导读
PhysMoE 通过按物理模态解耦 VLM 上下层——共享低层、上层分别路由至空间与语言专家，并用视觉编码器位置嵌入把每个坐标编码为单个 token，在 9 个坐标定位基准上平均提升 15.7%、吞吐提速 8.31 倍。
正文 · 原文
Abstract:Embodied Vision-Language Models (VLMs) are backbones for robotic planning and manipulation. However, for coordinate prediction in grounding tasks, existing architectures suffer from two distinct bottlenecks: discrete tokenization disrupts spatial continuity with high decoding latency, while standard point regression fails on multi-contact affordances and irregularly shaped targets. To resolve these challenges, we propose PhysMoE, introducing three core contributions: (1) sharing lower layers for multimodal context while routing upper layers to specialized spatial (pixel) and linguistic (text) experts by physical modality, eliminating gradient interference; (2) encoding each coordinate as a single token by reusing the visual encoder's learned positional embeddings, injecting 2D spatial priors to natively align coordinates with visual features and ease training; and (3) differentiable Signed Distance Field (SDF) supervision on pointing, accommodating affordances and irregular geometries without costly mask decoders. Compared to the base model, PhysMoE achieves a 15.7% average gain and an 8.31x throughput speedup across 9 coordinate-based grounding benchmarks (trajectory, affordance, and localization), while preserving multimodal understanding and reasoning across 11 text-based embodied benchmarks.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Robotics (cs.RO) |
| Cite as: | arXiv:2611.13000 [cs.CV] |
| (or arXiv:2611.13000v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13000 arXiv-issued DOI via DataCite |
Submission history
From: Ruimeng Liu [view email]
[v1]
Sun, 8 Nov 2026 16:00:02 UTC (969 KB)
来源：arXiv cs.AI · arxiv.org