Spike-EIFNet: Lightweight Spike-Driven Event-Image Fusion Network for Accurate and Efficient Semantic Segmentation.

Abstract

Semantic segmentation is critical for intelligent robotics to understand complex environments. While CNN-based models on RGB images achieve high performance, their accuracy drops in fast-motion or low-light scenes. Fortunately, event cameras, with high temporal resolution and low latency, offer robust perception in such challenging conditions. Many event-image fusion methods attempt to combine the complementary strengths of both modalities, but most adopt simple fusion strategies without considering intermodal correlations or designing computationally expensive architectures, resulting in degraded accuracy and high energy costs. To overcome these limitations, we propose a lightweight spiking neural network (SNN)-based event-image fusion network (Spike-EIFNet) that leverages the complementary strengths of multimodal fusion and energy-efficient spike-driven computation. In particular, to reduce computation cost for lightweight, Spike-EIFNet adopts a dual-branch SNN encoder to process events and images in parallel. Then, to improve the segmentation accuracy with enhanced feature interaction, we introduce a spike-driven cross-modal fusion (SCMF) module, consisting of a modality-aware fine-grained extraction (MFE) stage to capture dynamic cues from events and spatial details from images, followed by a cross-modal interaction and fusion (CIF) stage for effective feature alignment. Finally, a lightweight feature enhancement (LFE) module is proposed to further refine feature representations and facilitate deep-shallow feature fusion. Extensive experiments demonstrate that Spike-EIFNet achieves 67.34% and 58.09% mean intersection over union (mIoU) on the DDD17 and DSEC-Semantic datasets while consuming $72.83\times $ and $100.26\times $ less energy, respectively. Compared with ANN-based methods, Spike-EIFNet significantly reduces energy consumption; among SNN-based methods, it achieves the highest segmentation accuracy with a favorable accuracy-efficiency tradeoff. Code is available at: https://github.com/Chensyfighting/Spike-EIFNet.