CADSA: Content-Aware Dynamic Sparse Attention for Lightweight and Energy-Efficient Edge AI with Sub-Quadratic Complexity
TL;DR
CADSA is a content-aware dynamic sparse attention framework that lowers the quadratic complexity of self-attention to sub-quadratic for lightweight edge AI deployment, and fine-tuning strategy with gradient masking decreases the training memory overhead by 60%, allowing for practical on-device fine-tuning.
Abstract
This paper presents CADSA, a content-aware dynamic sparse attention framework that lowers the quadratic complexity of self-attention to sub-quadratic for lightweight edge AI deployment. A learnable token relevance scoring module dynamically prunes redundant attention paths by top-k selection in local neighborhoods, with complexity O(n·k·d). A variance-aware sparsity modulation mechanism is used for generative tasks, where the pruning ratio is adjusted according to the statistics of the local regions, focusing computational resources on detailed regions and simplifying uniform regions. Block-sparse execution kernels and bitmask compression reduce memory bandwidth by 32×, ensuring hardware efficiency. Experimental results show 2.1× inference speedup and 40% energy saving on Raspberry Pi 5, and 98.6% dense attention accuracy on ImageNet classification. On LSUN Bedrooms, CADSA achieves similar FID scores (12.7 vs. 12.3) and 2.4× faster sampling than diffusion models. The fine-tuning strategy with gradient masking decreases the training memory overhead by 60%, allowing for practical on-device fine-tuning.