跳到正文
arXiv cs.AI· Gabriele La Malfa, Nitay Alon, Emanuele La Malfa, Reuth Mirsky, Stefan Sarkadi·· 4 小时前AI 评分38

EMRD 基准:评估 VLM 在安全关键场景中的探索、建图、记忆与决策能力

Explore, Map, Remember, Decide: A Benchmark for VLM Readiness in Safety-Critical Scenarios

AI 导读

研究者提出 EMRD 评估框架,将 VLM 的空间推理拆为 Explore、Map、Remember、Decide 四阶段,用 18 项阶段指标在 16 栋建筑中评测 11 个开源与闭源 VLM。

正文 · 原文

View PDF HTML (experimental)

Abstract:Vision-Language Models (VLMs) are often deployed as embodied controllers for robotics, yet it remains unclear how well they can explore an environment, maintain an internal map, and ground safety-critical decisions in long-horizon sequential trajectories. In this work, we introduce EMRD, an evaluation framework that decouples spatial reasoning into four stages: Explore (what the agent observes), Map (what it records and where), Remember (what it retains), and Decide (whether its final choice is justified), scored across 18 stage-specific metrics. We use EMRD to evaluate eleven open- and closed-weight VLMs across 16 buildings under nominal conditions and three environmental stresses (darkness, smoke, and motion noise), where an embodied VLM explores unknown rooms, maps observed objects, and selects the safest gathering place. Open-weight models leave rooms unvisited in 95% of episodes and detect at most 51% of objects; closed-weight models explore more, though with an external memory (notepad), two of the three stop prematurely. Without external memory, most VLMs keep fewer of the objects they saw, especially in larger buildings. The start limits which rooms are explored, and open-weight models started in different rooms agree on the final room not significantly more often than chance. Moreover, these choices are biased by semantic priors: swapping a room's label is often enough to change the final choice. Finally, motion noise increases object position error (by 12-60% for closed-weight models), strong darkness lowers map recall by 28-31 percentage points, and dense smoke reduces seen objects by 25-39 points, while the choice among the rooms entered does not change significantly.
Subjects: Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA); Robotics (cs.RO)
Cite as: arXiv:2608.08077 [cs.AI]
  (or arXiv:2608.08077v2 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2608.08077

arXiv-issued DOI via DataCite

Submission history

From: Gabriele La Malfa [view email]
[v1] Sat, 8 Aug 2026 11:57:01 UTC (4,284 KB)
[v2] Tue, 17 Nov 2026 14:20:30 UTC (6,960 KB)

来源:arXiv cs.AI · arxiv.org