PNTAFormer: polarity-aware neighborhood token attention for spiking transformers
TL;DR
PNTAFormer, which incorporates PNT-Attention, enhances token attention by integrating two critical components: (1) 2D local neighborhood aggregation through depthwise convolution to capture spatial context, and (2) excitatory-inhibitory competition driven by membrane polarity in spiking neurons, enabling the model to effectively represent both positive and negative interactions between tokens.
Abstract
Spiking Transformers have recently adopted token-wise gating attention mechanisms to avoid the quadratic cost of dot-product attention. However, such token attention mechanisms discard critical relational information between tokens, which limits their expressive power. In this paper, we propose PNT-Attention, a novel attention mechanism designed to address the limitations of existing approaches in Spiking Transformers. PNTAFormer, which incorporates PNT-Attention, enhances token attention by integrating two critical components: (1) 2D local neighborhood aggregation through depthwise convolution to capture spatial context, and (2) excitatory-inhibitory competition driven by membrane polarity in spiking neurons, enabling the model to effectively represent both positive and negative interactions between tokens. Our approach maintains linear complexity and parameter efficiency, while significantly improving the expressive capability of the attention mechanism. Extensive experiments on both static and neuromorphic datasets demonstrate that PNTAFormer achieves significant performance improvements. On CIFAR10, it reaches 96.36% accuracy, outperforming Spikformer by 1.85% (94.51%), and on CIFAR100, it achieves 81.31%, surpassing Spikformer by 3.1% (78.21%). Additionally, PNTAFormer reduces the parameters from 9.32M to 6.74M, maintaining efficiency without sacrificing performance. On CIFAR10-DVS, PNTAFormer further achieves 84.7% top-1 accuracy with 1.5M parameters, demonstrating that the proposed attention mechanism remains highly effective even under strict model capacity constraints and event-based input settings. Moreover, under identical parameter budgets, PNTAFormer consistently outperforms QKFormer across both static and neuromorphic benchmarks, highlighting the importance of restoring token relational information in spike-based attention and demonstrating the effectiveness of polarity-aware local context modeling for Spiking Transformer design.