Marginal Contribution Evaluation of Federated Learning Clients With Generalization Gap Dynamics
TL;DR
Experimental results demonstrate that the proposed <inline-formula> <tex-math notation="LaTeX">$FedG^{2}$ </tex-math></inline-formula> outperforms widely-used baselines and achieves good performance on various heterogeneous settings.
Abstract
Federated learning (FL) has become the mainstream framework for training models on geographically dispersed edge devices. Owing to differences in sample availability and distribution, the quality of local data can vary substantially across clients. Many studies have proposed evaluating the contribution of clients’ local data using metrics such as local data proportion, local training loss, and gradient direction. However, they ignore a key training dynamics: scattered generalization gap. Some clients can fit to local minima quickly due to too poor sample quality or vast local training epochs while others being far from optima, leading to sub-optimal accuracy. In this paper, we focus on the generalization behaviour of clients during runtime, and propose the <inline-formula> <tex-math notation="LaTeX">$FedG^{2}$ </tex-math></inline-formula> algorithm. The <inline-formula> <tex-math notation="LaTeX">$FedG^{2}$ </tex-math></inline-formula> defines the utility dynamics of models with generalization gap and utilizes it to measure the marginal contribution of each client dynamically in each round. In addition, <inline-formula> <tex-math notation="LaTeX">$FedG^{2}$ </tex-math></inline-formula> incorporates the local update magnitudes into account to make updates more consistent with global objective. Experimental results demonstrate that <inline-formula> <tex-math notation="LaTeX">$FedG^{2}$ </tex-math></inline-formula> outperforms widely-used baselines and achieves good performance on various heterogeneous settings.