The results demonstrate that Fisher information provides an effective and unified mechanism for mitigating partition-induced covariate shift across both centralized and distributed learning.
Abstract
Covariate shift across training-data partitions biases model selection and parameter estimation in cross-validation, lifelong learning, and federated learning. We propose \textit{Partition-Induced Covariate-shift Correction} (\texttt{PIcsC}), a Fisher information-based regularization framework that mitigates distribution mismatch between data partitions and a reference distribution. \texttt{PIcsC} approximates partition divergence using the Fisher Information Matrix (FIM) and incorporates the resulting statistic as a regularizer during optimization. The same formulation applies to both centrally partitioned datasets (batches or cross-validation folds) and inherently distributed data (federated clients or decentralized nodes), requiring only partition-local gradient statistics rather than raw data. We further introduce a conditional adaptation mechanism that combines FIM shift with KL divergence to detect significant distribution shifts and activates regularization only when necessary. Experiments on more than 40 datasets demonstrate consistent improvements under both natural and synthetic covariate shift. On fragmented batch and fold settings, \texttt{PIcsC} reduces fragmentation-induced performance degradation by more than 20\% and 25\%, respectively. On seven federated learning benchmarks, it consistently outperforms FedAvg, FedProx, and SCAFFOLD by 3 -5 percentage points without requiring client-specific personalization. These results demonstrate that Fisher information provides an effective and unified mechanism for mitigating partition-induced covariate shift across both centralized and distributed learning.
The Conditional Jensen-Shannon Discrepancy (CJSD) is proposed, which proves a covariate-null property, a drift-mass law, a one-sided misspecification-control inequality, a one-sided misspecification-control inequality, and a fixed-measure metrization via an identifiability lemma.
This paper proposes a key notion: $\gamma^{*}\!$-concept shifts, and derive a general error bound unifying covariate and $\gamma^{*}\!$-concept shifts, which applies to broad loss functions, label spaces, and stochastic labeling and develops estimators for these shifts with concentration guarantees.
This work considers a multi-resolution setting in which individual-level data are available from a source population, while the target population is observed only through subgroup summaries, and proposes SAGE, a one-step estimator that updates a source-trained predictor using a gradient estimated from these summaries.
In many industrial applications, 1) tabular data is scarce and imbalanced and thus requires synthetic expansion; 2) input distributions drift between training and deployment (covariate shift); 3) validation sets often diverge from unseen test environments; or 4) standard generative models simply mimic outdated source d...
This article studies unsupervised transfer learning for a binary outcome, with labeled source data and unlabeled target covariates, and formulate this structure as group‐label shift (GLS) and pair it with a class‐specific exponential tilting model for the joint density ratio between the target and source populations.
Man-Li Cheng, Subha Maity, Qing-Long Tian et al.· Scandinavian Journal of Stat...· 0 citations
To commit to buying external data or participate in collaborative learning, one must decide whether the additional data will improve prediction enough to justify the cost. This comes with several challenges: (i) the decision often relies only on aggregated statistics available publicly, rather than individual-level dat...
Filip Kovacevic, Edwige Cyffers, Stefano Sarao Mannelli et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.