Large language models for evidence-based planning: Evaluating an SLR-RAG framework for knowledge synthesis of urban vacant land

Abstract

Urban challenges necessitate robust evidence synthesis, yet Large Language Models (LLMs) applied to urban studies face factual inconsistencies and a lack of domain-specific grounding. This paper proposes and evaluates a framework for RAG-enhanced LLMs by integrating up-to-date knowledge and policy-relevant evidence into model outputs. Urban Vacant Land (UVL) is selected as the case study, and a systematic literature review (SLR) is conducted to build a retrieval literature base. The results indicate that RAG significantly improves the accuracy of LLMs under structured automatic evaluation. However, although retrieval augmentation provides models with access to domain-specific evidence, its benefits for open-ended planning questions are not consistently reflected in expert ratings, particularly for consistency and creativity. The limited improvement can be attributed to the characteristics of the planning questions, the mismatch between textual information and the spatial data required for urban planning in current RAG pipelines, and potentially ineffective prompting that fails to elicit deeper reasoning. This study contributes to the field by elucidating the capabilities and limitations of LLMs and RAG in urban studies, highlighting that while text-only retrieval is insufficient for context-rich analysis, future advancements in spatially aware hybrid retrieval offer a promising pathway forward.