A multimodal remote sensing object detection model based on the Swin-MoE framework

Abstract

To address the substantial differences in imaging characteristics across sensors and the representation conflicts introduced by a shared backbone in multimodal remote sensing object detection, this paper presents a unified detection framework built on the Swin Transformer. A hierarchical Transformer is used as the shared feature extractor, and a grid-level sparse Mixture-of-Experts (MoE) routing module is embedded into the feed-forward network to allow local features from different modalities to select more suitable experts during inference. Unlike conventional multimodal fusion designs that mainly increase cross-modal interaction branches, and unlike coarse MoE strategies that route whole images or samples, the proposed method performs expert selection at the spatial grid level within a shared backbone. In this way, the model improves its ability to capture modality-specific patterns while retaining the efficiency of parameter sharing. Experiments on the SOI-Det benchmark, which combines SARDet-100K, DOTA, and DroneVehicle, show that the proposed framework consistently outperforms several representative baselines, reaching 51.82 mAP, 82.64 mAP@0.5, and 54.96 mAP@0.75, with a favorable trade-off between accuracy and complexity. Additional results show that joint multimodal training not only improves the overall performance of the unified model, but also brings gains to single-modality detection. The study further suggests that grid-level sparse MoE routing is effective in reducing cross-modal feature interference under a shared backbone. Overall, the results indicate that combining hierarchical Transformer-based shared representations with fine-grained expert routing is a practical solution for unified multimodal remote sensing object detection.