This work introduces a pioneering exploration of Self-Supervised Learning (SSL) within the SNN, and proposes a novel Spiking Self-Attention (SSA) and Spiking Transformer (Spikformer) that achieves 80+% accuracy on ImageNet.
Abstract
Spiking Neural Networks (SNNs), known for their biologically plausible architecture, face the challenge of limited performance. The self-attention mechanism, which is the cornerstone of the high-performance Transformer and also a biologically inspired structure, is absent in existing SNNs. To this end, we explore the potential of leveraging both self-attention capability and biological properties of SNNs, and propose a novel Spiking Self-Attention (SSA) and Spiking Transformer (Spikformer). The SSA mechanism eliminates the need for softmax and captures the sparse visual feature employing spike-based Query, Key, and Value. This sparse computation without multiplication makes SSA efficient and energy-saving. Further, we develop a Spiking Convolutional Stem (SCS) with supplementary convolutional layers to enhance the architecture of Spikformer. The Spikformer enhanced with the SCS is referred to as Spikformer V2. To train larger and deeper Spikformer V2, we introduce a pioneering exploration of Self-Supervised Learning (SSL) within the SNN. Specifically, we pre-train Spikformer V2 with masking and reconstruction style inspired by the mainstream self-supervised Transformer, and then finetune the Spikformer V2 on the image classification on ImageNet. Extensive experiments show that Spikformer V2 outperforms other previous surrogate training and ANN2SNN methods. An 8-layer Spikformer V2 achieves an accuracy of 80.38% using 4 time steps, and after SSL, a 172M 16-layer Spikformer V2 reaches an accuracy of 81.10% with just 1 time step. To the best of our knowledge, this is the first time that the SNN achieves 80+% accuracy on ImageNet. The code will be available at Spikformer V2.
This work pioneers the synergistic integration of SNNs into Transformer architectures for LLIE, establishing a compelling pathway toward powerful, energy-efficient low-level vision on resource-constrained platforms.
Hongzhi Wang, Xiubo Liang, Jinxing Han et al.· Advances in Neural Informati...· 0 citations
A Spiking Neural Network (SNN) is a kind of brain-inspired and event-driven network, which is becoming a promising energy-efficient alternative to Artificial Neural Networks (ANNs). In recent years, SNN methods have been successfully applied in the fields of electromagnetic signal processing and image signal processing, particularly in application scenarios that require low energy consumption. However, the performance of SNNs by direct training is far from satisfactory. In this paper, we study a novel learning method named SAD-SNN (Spatial-Activation Distillation for Spiking Neural Networks), which utilizes the ANN model to guide the SNN model learning. Unlike prior works that rely on element-wise feature alignment approaches, SAD-SNN aligns spatial-activation maps at different resolutions of the teacher and student networks. Specifically, we introduce a direct alignment approach, which defines a spatial-activation loss and normalizes the representation vectors of ANN and SNN, to alleviate the unexpected precision loss. This enables the knowledge of teacher ANNs to be effectively transferred to train student SNNs. On three image classification datasets, our proposed SAD-SNN outperforms other SNN training methods no matter whether homogeneous or heterogeneous teacher ANNs are used. Furthermore, we apply SAD-SNN to the electromagnetic signal detection task, demonstrating strong generalization ability and superior performance. In conclusion, the experimental results on various tasks and SNN architectures demonstrate that our method is a general and effective solution that significantly improves the learning of student SNNs with only two time steps.
Chongxiao Qu, Qian Zhang, Chenxiao Dou et al.· Italian National Conference...· 0 citations
Keyword spotting (KWS) systems based on Spike Neural Networks (SNNs) offer a promising solution for always-on voice interfaces. However, achieving a favorable trade-off between computational footprint and recognition accuracy remains challenging for resource-constrained edge devices. This paper proposes a lightweight convolutional spiking neural network (CSNN) for KWS that combines a streamable Mel-to-Spike encoder, a convolutional spiking feature extractor, and a delay-aware classification module that uses learnable synaptic delays. The proposed encoder adopts streaming frame-by-frame encoding to convert speech features into sparse spike trains, while the delay-aware classifier jointly optimizes synaptic weights and temporal delays for enhanced spatiotemporal evidence aggregation. Experiments on the Google Speech Commands V1 and V2 (GSC-V1 and GSC-V2), Heidelberg Digits (HD), and Chinese Mandarin Keyword (CMK) datasets show mean test accuracies of 94.37%, 92.87%, 99.10%, and 95.60%, respectively. The proposed method uses only 64.05 K and 68.14 K learnable parameters for the 12-class and 20-class classification, while maintaining strong robustness to additive noise. These results indicate that the proposed CSNN achieves a favorable algorithm-level balance among accuracy, compactness, and noise robustness for KWS.
Xiaohuan Li, Yi Liu, Libo Zheng· Italian National Conference...· 0 citations
Spiking Neural Networks (SNNs) enable event-driven computation with sparse activations, but building multimodal Transformers on SNNs is hindered by unstable training in deep spiking stacks and the mismatch between dense softmax attention and spike-based communication. We propose SMM Transformer, an SNN-based multimodal Transformer framework that combines (i)PLMP, a Parallel LIF with Multistage Learnable Parameters neuron and a tailored P-STBP algorithm for stable deep SNN training, (ii) SMSA, an attention-inspired spike-driven token-mixing module that replaces dense pairwise softmax attention with channel-wise spike co-activation and self-compensation, and (iii)SMoE, a spiking mixture-of-experts module for modality-aware fusion. Across visual and multimodal benchmarks, SMM Transformer achieves competitive accuracy compared to ANN baselines. Under a standard MAC/AC arithmetic model, SMSA reduces the estimated operator-level compute energy of the attention module by up to 97%, while whole-model profiling shows more moderate but consistent efficiency gains.
Xiubo Liang, Jinxing Han, Yuke Li et al.· 0 citations
HazeSpikeMamba is proposed, a compact dehazing framework that combines a spiking-inspired local path and an attentive state-space global path in a multi-scale U-Net that consistently improves BRISQUE and NIMA on RTTS, URHI, and HSTS.
Haoran Liu, Huibin Li, Mingzhe Liu et al.· 0 citations
This work proposes the use of ternary spike neurons, which enhance the information-carrying capacity in the spiking neural rendering model, and introduces Spik-NeRF (Spiking Neural Radi-ance Fields with Ternary Spike), which achieves rendering performance comparable to ANN-based NeRF models.
Gang Wan, Qinlong Lan, Zihan Li et al.· Neural Information Processin...· 1 citation
A new method for surgically removing training examples from a model reveals that as datasets grow, the link between what a model learns and what it produces dissolves.
