Transmission-Efficient Federated Learning via Chessboard Convolutional Kernel

Abstract

Reducing the communication cost to alleviate the contradiction between the limited bandwidth and huge transmitted parameters in Federated Learning (FL) is a persisting challenge. Since the transmission parameters obtained by vanilla convolution primarily rely on the product of kernel and channels, recent works (e.g., Fed-KGF) reduce the transmission parameters by reducing the number of channels. While they ignore the impact of kernel elements on transmission efficiency. In this paper, we focus on reducing the density of convolutional kernel and innovatively propose a novel convolutional kernel, named Chessboard Convolutional Kernel (i.e., <bold>Fed-ChessConv</bold>), to further significantly reduce the transmission parameters while maintaining the performance of the global model in FL. Given a kernel with <inline-formula><tex-math notation="LaTeX">$n\times n$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>n</mml:mi><mml:mo>×</mml:mo><mml:mi>n</mml:mi></mml:mrow></mml:math><inline-graphic xlink:href="li-ieq1-3713384.gif"/></alternatives></inline-formula> size (i.e., <inline-formula><tex-math notation="LaTeX">$n^{2}$</tex-math><alternatives><mml:math><mml:msup><mml:mi>n</mml:mi><mml:mn>2</mml:mn></mml:msup></mml:math><inline-graphic xlink:href="li-ieq2-3713384.gif"/></alternatives></inline-formula> elements within this kernel), we retain elements that are at even-row-even-column and odd-row-odd-column positions and discard the rest, forming a chessboard-like convolutional kernel with a size of <inline-formula><tex-math notation="LaTeX">$\lceil \frac{n^{2}}{2} \rceil$</tex-math><alternatives><mml:math><mml:mrow><mml:mo>⌈</mml:mo><mml:mfrac><mml:msup><mml:mi>n</mml:mi><mml:mn>2</mml:mn></mml:msup><mml:mn>2</mml:mn></mml:mfrac><mml:mo>⌉</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="li-ieq3-3713384.gif"/></alternatives></inline-formula>. Specifically, chessboard convolutional kernel extracts data features at the first step. Because of a chessboard convolutional kernel with size of <inline-formula><tex-math notation="LaTeX">$\lceil \frac{n^{2}}{2} \rceil$</tex-math><alternatives><mml:math><mml:mrow><mml:mo>⌈</mml:mo><mml:mfrac><mml:msup><mml:mi>n</mml:mi><mml:mn>2</mml:mn></mml:msup><mml:mn>2</mml:mn></mml:mfrac><mml:mo>⌉</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="li-ieq4-3713384.gif"/></alternatives></inline-formula> rather than <inline-formula><tex-math notation="LaTeX">$n\times n$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>n</mml:mi><mml:mo>×</mml:mo><mml:mi>n</mml:mi></mml:mrow></mml:math><inline-graphic xlink:href="li-ieq5-3713384.gif"/></alternatives></inline-formula>, not all features are involved in extraction at the first step. However, those uninvolved features would be supplemently extracted at the second step after sliding the window for one stride. With the window sliding stride by stride on images, chessboard convolutional kernel extracts all features as the vanilla one does, with a stride of 1, while the number of the transmitted parameters is reduced by approximately 50%. We deploy one network with chessboard convolutional kernel to each client and the server in FL, and we transmit the updated parameters from clients to server. The performance of our chessboard convolutional kernel and the global model in our <bold>Fed-ChessConv</bold> have been theoretically analyzed. Experimental results on both non-Independent and Identically Distributed (non-IID) and IID scenarios for image classification and object detection tasks demonstrate that our <bold>Fed-ChessConv</bold> outperforms SOTA models. <bold>Fed-ChessConv</bold> achieves 3.4% (3.96%) higher classification accuracy and roughly 41.61% (96.91%) fewer transmission parameters on CIFAR-100, and presents similar detection performance with 42.51% (93.61%) fewer transmission parameters than the recent Fed-KGF (Fed-LTP) on COCO2017 datasets.