TFVP-MLLM: Training-Free Visual Prompting Multimodal Large Language Model for Remote Sensing Image Classification

Abstract

The multimodal large language models (MLLMs) have achieved breakthroughs in remote sensing applications because of their powerful cross-modal understanding, enabling effective alignment between linguistic descriptions and visual data. However, the existing MLLM-based approaches still suffer from challenges, such as inaccurate region localization, limited domain adaptability, and high computational costs. To overcome these limitations, we propose a training-free visual prompting MLLM called TFVP-MLLM, which is built on visual prompting guidance and a scoring feedback mechanism to achieve high-quality remote sensing image classification under low resource constraints. Our TFVP-MLLM is composed of the category proposal generator (CPG) and the score feedback refiner (SFR). The CPG is responsible for generating high-quality candidate categories by integrating the proposed geographic priors injection module (GPIM) and hybrid-granularity region attention loss (HGRAL). Meanwhile, SFR uses an iterative generate and score process to enhance the accuracy and reliability of the candidate categories. As two critical components of CPG, GPIM incorporates prior geographic knowledge to significantly improve the model’s domain adaptation, while HGRAL optimizes visual token representations with several frozen modules to reduce computational requirements. To validate the effectiveness of the proposed TFVP-MLLM, the comprehensive experiments are conducted on three benchmark datasets UCM, AID, and DIOR-RSVG. For the scene classification task, TFVP-MLLM achieves the state-of-the-art performance among training-free multimodal approaches with overall classification accuracies of 97.8% and 82.4% on the UCM and AID datasets, respectively. For the regional object classification task, TFVP-MLLM achieves remarkable scores of 92.4% semantic similarity (SS) and 88.6% semantic intersection over union (SIOU) on the DIOR-RSVG dataset, which surpasses the other training-free methods and even achieves comparable results against supervised approaches. In addition, we perform a series of ablation studies to confirm the effectiveness of each module. The source code is available at https://github.com/Camellia86/TFVP-MLLM