Information Bottleneck with Differential Privacy for Federated Learning

Abstract

Federated learning (FL) offers a promising approach to training machine learning models on decentralized data sources without direct data sharing, thus addressing privacy concerns. However, FL is still susceptible to privacy breaches and suffers from significant information loss during model aggregation, a phenomenon addressed by the information bottleneck (IB) principle. This paper proposes a novel framework that integrates the IB technique with differential privacy (DP) within the FL setting. We formulate the problem as a constrained optimization, minimizing information loss while simultaneously satisfying DP guarantees. Our approach utilizes a compressed representation of local data, learned through an IB objective, and introduces noise to protect individual data points, ensuring privacy. The core contribution lies in the synergistic combination of these two techniques, leading to enhanced privacy protection and improved model accuracy compared to standard FL. We demonstrate the effectiveness of our framework through a theoretical analysis and outline potential implementation strategies. The primary goal is to achieve a balance between model performance and privacy preservation, a critical aspect often overlooked in current FL methodologies. The theoretical framework provides a foundation for future research and practical deployment in privacy-sensitive applications.