Spatiotemporal Focal Modulation Networks for Efficient Human Action Recognition

Abstract

The most common approaches to video action recognition have been using computationally expensive 3D Convolutional Neural Networks (3D CNNs) or using transformers, relying on the self-attention mechanism. Video Transformers are good at modeling long-range spatiotemporal relations; yet they are not good enough for real-time application because of the quadratic complexity (in order of $\mathbf{N}^{\mathbf{2}})$ of self-attention. In this paper, we develop a new hybrid model that consists of a 3D Convolutional Neural Network as the backbone and a 1D Focal Modulation Transformer as the encoder. Our approach first extracts local spatiotemporal features using pre-trained R(2+1)D or 3D ResNet-18 as backbone, and then re-shapes them to a sequence of tokens in order to model context efficiently. Unlike traditional self-attention, our approach models context using a hierarchical focal modulation approach which involves multi-scale depth wise convolutions to extract both local and global context. Our proposed model has significant computation savings, and also retains strong inductive biases for structured learning by substituting self-attention with the local-to-global modulation. Our model obtains 96.13% peak validation accuracy on UCF101 standard human action recognition dataset.