Exploring Latency Hiding Strategies for GPU GEMM Using Pipelined Multiword Arithmetic
Abstract
Matrix multiplication is at the heart of many applications in AI, scientific computing, and quantum simulations, for which GPUs have been widely adopted due to the immense compute power and hardware acceleration they provide for such computations. In this paper, we explore the performance profile of computing dense matrix–matrix multiplications (GEMM) on GPUs by decomposing the operation into several cascading lower-precision GEMMs and exploiting the then-available overlap between data transfers and mixed-precision compute and measure its performance when compared to standard monolithic or tiled GEMMs. The proposed implementation exploits the high performance of low-precision matrix multiply-accumulate (MMA) operations on modern GPUs and their multi-stream execution capabilities to overlap multiple low-precision GEMMs with data transfers.