FedAD: An Adaptive Block Dropping Approach for Federated Learning on Resource-Constrained Devices

Abstract

—In a federated learning environment with heterogeneous, limited and time-varying computational resource capabilities, it is challenging to update deep neural network in real-time. To address this challenge, we propose FedAD, an adaptive block-dropping approach for federated learning on resource-constrained devices. FedAD dynamically adjusts the size of the local training model in real-time by dropping blocks based on the device’s varying resource availability. This is controlled by a policy network, which determines the drop probability for each block depending on data inputs. Using the policy network’s output, FedAD generates a drop policy that decides which blocks in the neural network are dropped during local training. Applying this method, each device can independently generate the drop policy that dynamically adapts to the input and real-time available resources. Extensive experiments demonstrate that FedAD converges faster and outperforms state-of-the-art methods in resource-constrained settings, performing on par with or better (between 10% to 50%) than baselines in resource-rich environments. Additionally, FedAD demonstrates strong adaptability to real-time resource fluctuations and high flexibility in accommodating heterogeneous resources.