arXiv cs.AI· Xinyi Zheng, Yunze Liu, Chi-Hao Wu, Fan Zhang, Hao Zheng, Wenqi Zhou, Walterio W. Mayol-Cuevas, Junxiao Shen·· 12 小时前AI 评分38
SpatialMem：面向语言定位与问答的度量对齐长时程视频记忆系统
SpatialMem: Metric-Aligned Long-Horizon Video Memory for Language Grounding and QA
AI 导读
SpatialMem 是一个融合 3D 几何、语义与语言的可查询记忆系统，仅用第一视角 RGB 视频即可重建度量尺度的室内场景并支持空间关系推理与问答。在 VSI-Bench（288 个视频、5,130 道题）上以纯 RGB 取得 37.5 总分，房间尺寸子项为 41.4，而同等规模帧模型仅 24.4。
正文 · 原文
Abstract:SpatialMem is a memory-based system which integrates 3D geometry, semantics, and language with one another in one combined, queryable representation. Starting with casually recorded egocentric RGB video, SpatialMem recreates metrically scaled interior scenes, recognizes structural 3D anchors (walls, doors, windows) as the first-layer scaffold, and associates open-vocabulary object nodes with evidence patches, visual embeddings, and two-layer descriptions to 3D coordinates, enabling spatial-relation reasoning, language-directed retrieval, and QA without special sensors. We test at scale on public benchmarks. On VSI-Bench ($288$ videos, $5{,}130$ questions) SpatialMem scores $37.5$ overall with RGB alone, and its margin is precisely where a metric memory should earn it (room size $41.4$ vs.\ $24.4$ with a same-size frame model). On the episodic-memory task of OpenEQA ($953$ questions) the memory serialised to text is lower than the blind control, but as evidence-frame retrieval it recovers frame-level accuracy, matching uniform sampling at an equal eight-frame budget and pulling ahead as the budget tightens, with every answer grounded in linked 3D evidence. On $12$ ScanNet scenes evaluated using instance-level official annotations, the anchor-first hierarchy assigns the retrieved object to the proper structural parent where open-vocabulary 3D scene graphs essentially cannot ($0.24$ compared to $0.01$ with matched input), a capability difference instead of an overall win. The queryable index has a cost of $14$\,KB per scene without the linked evidence frames, and we make available the suites, memories and code.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2601.14895 [cs.CV] |
| (or arXiv:2601.14895v4 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2601.14895 arXiv-issued DOI via DataCite |
Submission history
From: Xinyi Zheng [view email]
[v1]
Wed, 21 Jan 2026 11:32:24 UTC (6,450 KB)
[v2]
Thu, 5 Mar 2026 22:39:31 UTC (7,055 KB)
[v3]
Thu, 22 Oct 2026 10:53:26 UTC (1,239 KB)
[v4]
Fri, 23 Oct 2026 14:47:39 UTC (4,434 KB)
来源：arXiv cs.AI · arxiv.org