Skip to content

USAI-Quant: A Quantitative Reasoning Benchmark for Vision-Language Models in Built Environments

Sep 2026 · 0 citations · 39 references
Computer Science

TL;DR

The USAI-Quant benchmark is developed, the first benchmark designed to quantitatively evaluate VLM's reasoning capabilities on built environment metrics via remote sensing imagery, and reveals that current state-of-the-art models consistently fall short on numeric reasoning tasks.

Abstract

Large vision-language models (VLMs) have emerged as a powerful paradigm for urban and spatial AI. However, current state-of-the-art large VLMs still struggle with quantitative reasoning on remote sensing imagery. Existing benchmarks and algorithms are predominantly based on qualitative Visual Question Answering (VQA), providing limited insights into the quantitative reasoning capabilities of VLMs for built environment metrics. To address this gap, we develop Quantitative Urban and Spatial AI benchmark (USAI-Quant), the first benchmark designed to quantitatively evaluate VLM's reasoning capabilities on built environment metrics via remote sensing imagery. USAI-Quant is curated from the 335 largest U.S. cities, aligning high-resolution remote sensing images with quantitative built environment metrics. We then evaluate both general-purpose and remote sensing VLMs (RS-VLMs) by applying VQAs to tens of built environment metrics across three complexity levels. Our results reveal that current state-of-the-art models consistently fall short on numeric reasoning tasks. We further conduct in-depth analyses across models, question types, and geographic locations, uncovering insights into performance variability and task-specific challenges.

View source

Similar papers

Review Open access Aug 2026

Spatial intelligence in vision-language models: a comprehensive survey

This survey provides a comprehensive and unified overview of recent advances in spatial intelligence for VLMs, summarize core concepts behind spatial reasoning in VLMs, analyze why spatial failures occur, and organize existing solutions into a clear framework spanning prompting-based techniques, model improvements, exp...

Di-Sheng Liu, Tuo Liang, Zhe Hu et al. · 9 citations
Preprint Aug 2026

AeroGround: A Comprehensive Benchmark for Aerial-Ground Collaborative Reasoning

By systematically revealing the strengths and limitations of existing models in aerial-ground collaborative reasoning, AeroGround provides a foundation for developing more capable aerial-ground collaborative embodied intelligence systems.

Shenghong Yi, Lin Zhang, Muzian Li et al. · 0 citations
Preprint Oct 2026

From Sight to Foresight: Predictive Spatial Reasoning in Vision-Language Models

SpatialMind is introduced, a metric-scale VLM for spatial reasoning and future prediction that substantially outperforms both general and spatially specialized models on the authors' benchmark while achieving competitive zero-shot performance on VSI-Bench, OSI-Bench, and VLM4D.

Fei-Ran Wang, Xiao-Qi Wang, Zi-Wei Li et al. · 0 citations
Book Open access Oct 2026

MapQA: A Map-Question-Answering Benchmark for Visual Language Model Reasoning

Maps are central to how humans make sense of the world, from navigation and environmental monitoring to military planning and historical interpretation. Yet despite rapid progress in large multimodal models (LMMs), these systems continue to struggle with interpreting maps – an essential skill for visual reasoning that...

Christian M. Arnold, Andrew Alini, Abdulrahman Alabdulkareem et al. · 1 citation
Preprint Aug 2026

MG-VQA: Manipulation Grounded Visual Question Answering with VLMs

Vision-language models (VLMs) have shown promising spatial reasoning capabilities from static visual inputs, where the evidence needed to answer a question is available in the provided views. However, in cluttered environments, answer-relevant evidence may be occluded rather than absent: an object may lie beneath a pil...

Vineet Bhat, Mikaela Angelina Uy, Si-Yi Chen et al. · 0 citations

Validating, Not Sampling: Region-Level Robustness of Vision-Language and Vision-Language-Action Models

This VLM and VLA robustness validation shows that robustness is mostly dependent on the perturbation type, rather than the model, and that VLMs are more robust to large camera rotations than VLAs, and that robustness depends more on model family than on model size.

Bogdan Aron, Christopher Brix, Benedikt Brückner et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Oct 28, 2026

Making machines that can listen

Professor Paris Smaragdis develops computers with superhuman hearing for applications like speech recognition, music understanding, and environmental monitoring.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.