Efficient Binary ViTs through Hardware-Software Co-Design: A Power-Memory-Compute Optimization Framework

Abstract

Vision Transformers (ViTs) achieve state-of-the-art performance in computer vision, but are limited by high computational costs and data-intensive training. Thus, on resourceconstrained and edge devices, where power, memory, and compute budgets are severely limited, the practical deployment of ViTs becomes a significant challenge. To address this, model compression techniques like binarization have shown promise in reducing complexity while maintaining accuracy. In this work, a hardware-software co-design framework for Binarized Vision Transformers (BiViT), a compressed ViT model that features binarized weights and activations, is presented. The proposed framework is implemented on Xilinx Artix-7 FPGA and synthesized on 90nm ASIC hardware platforms, operating at a frequency of 200 MHz. The implementation reduces memory requirements by $27.5 \times$ and computational complexity (FLOPs) by nearly $196\times$, while preserving comparable accuracy. The viability of this approach is confirmed by its successful validation on both ViT-B and Swin-T models, demonstrating a top-1 accuracy of 96.59% in Swin-T.