Block Floating Point Based Matrix Multiplication Accelerator with Hardware Efficient Exponent Alignment

Abstract

Transformer models are popular in their usage for a wide range of applications. However they are computationally expensive in comparison to other deep learning models such as Convolutional Neural Networks (CNNs) and benefit g reatly in performance from hardware acceleration and reduced precision arithmetic. This paper demonstrates the design and performance evaluation of a matrix multiplication accelerator that uses 8-bit Block Floating Point (BFP8) arithmetic. Through the use of a novel exponent alignment technique and hardware aware design, the accelerator makes optimal use of the on-board resources in the Xilinx ZCU104 evaluation board on which it was implemented. The accelerator provides a throughput of 175 GOPS at 200MHz based on simulation results for matrix multiplication workloads. The power utilization for the proposed design was 4.208 W. Evaluation of the accelerator performance for the Llama 3.2 1-B parameter model with a hybrid CPU-Accelerator setup yielded an estimated theoretical kernel compute speedup gain of $110.7\times$ over CPU only inference.