Sparsity-Aware Vision Transformer Network for Efficient Event-Based Action Recognition

Abstract

Event cameras can capture human motion with low latency and high dynamic range, providing an emerging sensing paradigm for action recognition. A large body of recent work converts event data into a sequence of dense frames and feeds them into a video classification model for prediction. Although the frame-based methods take advantage of pre-trained backbones from the image domain to achieve high accuracy, most existing works sacrifice the sparsity of events due to dense processing, thereby increasing model complexity and latency. To boost the efficiency of frame-based methods while fitting the sparse nature of events, we propose the Sparsity-Aware Vision Transformer Network (SVTNet) with two novel event-oriented designs. Instead of direct event-to-frame conversion, we introduce an event representation method named Progressive Cumulative Event Representation (PCER) that adaptively integrates spatiotemporal cues across multiple temporal slices to better preserve fine-grained information. To leverage the spatial sparsity of events to accelerate model inference, we present the Motion Prior-Guided Token Sparsification Module (MPTS) that prunes redundant visual tokens hierarchically based on the joint motion-semantic importance. Comprehensive experiments show that SVTNet achieves state-of-the-art accuracy on multiple benchmark datasets and maintains much lower model complexity than existing frame-based methods.