MESSY STREETS: A Benchmark for Geocoding Real-World Addresses

Abstract

We introduce MESSY STREETS, a benchmark for evaluating geocoders on verbatim web addresses, with existence verification and controlled measurement of surface-form divergence. Unlike conventional benchmarks based on clean or synthetically perturbed addresses, MESSY STREETS contains addresses whose surface forms diverge from canonical representations and whose components may be missing, repeated, malformed, or incomplete. The benchmark is constructed from the December 2024 Web Data Commons corpus, with reference locations established from OpenAddresses or OpenStreetMap. The strongest commercial geocoders outperform open-source systems by up to 49 percentage points in recall. This gap is driven primarily by differences in candidate return rates on non-canonical addresses; once a candidate is returned, positional accuracy is broadly comparable across systems. Non-canonical surface form alone accounts for up to 25 percentage points of recall loss. Examining Nominatim's query-processing pipeline, we show that its conjunctive matching lets a single unrecognised token zero an otherwise valid query. The results demonstrate that geocoder choice is a consequential design decision for applications processing noisy address data, and that normalisation and preprocessing could substantially narrow the gap between open-source and commercial geocoders.