arXiv cs.AI· Peter A. Massih, Eric Cosatto, Martin Renqiang Min·· 13 小时前AI 评分31
SQuID：卫星影像定量空间推理基准
SQuID: A Benchmark for Quantitative Spatial Reasoning on Satellite Imagery
AI 导读
研究团队推出 SQuID 基准，包含 587 张卫星影像上的 2,000 道题，覆盖 24 种题型和三个难度层级，答案由人工标注的分割掩码或人类共识计算得出。在 15 个受测系统中，最佳零样本端到端 VLM 仅达 28.1%，遥感专用 VLM 得分更低；其提出的 QASR 基线达到 42.0%，改用参考掩码后升至 82.6%，显示分割是主要误差来源。
正文 · 原文
Abstract:Vision-Language Models (VLMs) describe satellite images fluently but struggle with the quantitative spatial reasoning that remote sensing (RS) analysts perform routinely, such as counting buildings, measuring forest area in hectares and computing distances in meters. We introduce SQuID (Satellite Quantitative Intelligence Dataset), a benchmark of 2,000 questions over 587 images from four sources at 0.3 to 1.0 m ground sampling distance, covering 24 question types in three difficulty tiers. Answers are computed from curated segmentation masks or, for 50 questions, from human consensus, and numeric answers are scored against tolerance ranges informed by human annotations. SQuID also releases the reference masks behind its answers, so that modular systems can be evaluated using reference rather than predicted masks, and 75,687 training question-answer pairs on disjoint images (SQuID-Train). Among 15 evaluated systems, the best zero-shot end-to-end VLM reaches 28.1%, and RS-specialized VLMs score lower. As a baseline, we propose QASR (Quantitative Agentic Spatial Reasoning), in which an LLM writes a program that segments the image and measures the masks with geometric operators. QASR reaches 42.0%. On the 1,950 mask-derived questions, running its programs on the reference masks instead of predicted masks raises accuracy from 42.2% to 82.6%, so segmentation is its main source of error. Fine-tuning Qwen2.5-VL-7B on SQuID-Train raises its accuracy from 16.1% to 40.7%, close to QASR. Multi-condition questions (Tier 3) remain the hardest in every setting, at 62.0% for QASR with reference masks and 13.2% after fine-tuning. These results leave clear room for end-to-end models that learn metric spatial reasoning, and SQuID provides the data to train and evaluate them.
| Comments: | Revised version with updated title, author list, manuscript, and experiments. Submitted to NeurIPS 2026 Evaluations & Datasets Track. Introduces the SQuID benchmark and QASR baseline for quantitative spatial reasoning on satellite imagery. Dataset DOI: https://doi.org/10.57967/hf/7565. Code and data are publicly available |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2601.13401 [cs.CV] |
| (or arXiv:2601.13401v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2601.13401 arXiv-issued DOI via DataCite |
Submission history
From: Peter A. Massih [view email]
[v1]
Mon, 19 Jan 2026 21:14:34 UTC (11,425 KB)
[v2]
Tue, 27 Oct 2026 14:45:28 UTC (13,248 KB)
来源：arXiv cs.AI · arxiv.org