S2MM: Scalable FPGA Acceleration of Secure Matrix Multiplication with Homomorphic Encryption
TL;DR
S2MM is presented, the first scalable FPGA-based accelerator designed for HE MM, and a novel datapath for Homomorphic Linear Transformation (HLT), the dominant workload in HE MM is proposed, enabling fine-grained on-chip data reuse and substantially reducing both off-chip memory traffic and on-chip memory demand.
Abstract
Homomorphic Encryption (HE) enables secure computation on encrypted data, addressing privacy concerns in cloud computing. However, the high computational cost of HE operations, particularly matrix multiplication (MM), remains a major barrier to its practical deployment. Accelerating Homomorphic Encrypted MM (HE MM) is crucial for applications such as privacy-preserving machine learning. In this paper, we present S2MM, the first scalable FPGA-based accelerator designed for HE MM. We show that HE MM is fundamentally bottlenecked by excessive off-chip memory traffic when operating with practical HE parameter sets. To analyze this bottleneck, we develop a cost model that characterizes the on-chip memory requirements of ciphertexts during HE MM, revealing the scalability limits of CPU-based implementations. Guided by the analysis, we propose a novel datapath for Homomorphic Linear Transformation (HLT), the dominant workload in HE MM. The proposed datapath enables fine-grained on-chip data reuse, substantially reducing both off-chip memory traffic and on-chip memory demand. To further improve scalability, S2MM supports tiled HE MM, removing the matrix size limitation imposed by the HE encoding capacity and enabling efficient computation on large matrices. In addition, the architecture is highly configurable, adapting to diverse matrix sizes and HE parameter sets while maximizing throughput within FPGA resource constraints. We also introduce a systematic configuration methodology to determine optimal design parameters for efficient deployment across varying workloads and platforms. We implement S2MM on an Alveo U280 and conduct comprehensive evaluations across a wide range of matrix sizes, shapes, and HE parameters. Experimental results demonstrate that S2MM achieves average speedups of 219× and 37.4× over state-of-the-art CPU and GPU implementations, respectively. We further evaluate S2MM in an application-level encrypted CNN inference pipeline, achieving a 6.08× end-to-end speedup by accelerating the HE MM-dominated fully connected layers.