VLM4S: A Vision‐Language Model Framework for Geographically and Environmentally Informed Soundscape Assessment

Abstract

Sound environments shape urban experience, yet conventional noise monitoring relies on isolated sensors lacking semantic context and spatial scalability. In this study, we develop a Vision‐Language Model for Soundscape (VLM4S), an exploratory, sensor‐free framework for inferring acoustic conditions from street‐view imagery. This framework addresses three tasks: identifying visible sound sources, predicting sound pressure levels (SPL), and classifying sound types (Natural/Mechanical/Human‐made). Utilizing a small‐scale experimental dataset of 200 co‐located training pairs, 20 pilot validation samples, and 40 out‐of‐sample test cases spanning eleven cities globally, we evaluate whether a vision‐language foundation model can utilize its pre‐trained visual knowledge to estimate real‐world environmental sound levels. The fine‐tuned model achieves a Mean Absolute Error (MAE) of 5.4 dB on the pilot validation set. Ablation analysis reveals that integrating street‐view imagery and Mel‐spectrograms with a Chain‐of‐Thought (CoT) prompt achieves the lowest error ( MAE=5.9dB$$ \mathrm{MAE}=5.9\ \mathrm{dB} $$ ), outperforming pure visual scrutiny ( MAE=12.4dB$$ \mathrm{MAE}=12.4\ \mathrm{dB} $$ ) and direct prediction without CoT ( MAE=9.8dB$$ \mathrm{MAE}=9.8\ \mathrm{dB} $$ ). External comparisons against United States Department of Transportation annual noise maps across 4000+ Manhattan locations reveal a moderate spatial correlation ( r=0.504$$ r=0.504 $$ , MAE=9.1dB$$ \mathrm{MAE}=9.1\ \mathrm{dB} $$ , Bias=+6.7dB$$ \mathrm{Bias}=+6.7\ \mathrm{dB} $$ ). While output discretization limits this generative approach for strict regulatory use, the framework provides a highly sample‐efficient baseline for exploratory soundscape assessment and hotspot identification in data‐scarce regions.