This paper derives a closed-form optimal linear operator from a small calibration set to reconstruct the activation discrepancy introduced by the pruned layers, and shows that this solution corresponds to the unconstrained optimum of the alignment objective, whereas LinearPatch is restricted to a fixed basis symmetric...
Vincent-Daniel Yun, Junhyuk Jo, Sai Praneeth Karimireddy et al.· arXiv.org· 3 citations
Federated learning (FL) is a communication-efficient distributed learning paradigm. However, client drift remains one of the most critical challenges, hindering the efficient training of a global model. In this study, we propose a novel latent information sharing scheme that directly mitigates data heterogeneity across...
Seung-Jun Lee, Ensieh Khazaei, Dimitrios Hatzinakos et al.· 1 citation
Federated continual learning must integrate new tasks over time without losing earlier-task knowledge. Most existing methods attach an anti-forgetting mechanism to the client-trained, server-aggregated loop of federated learning, which holds back new learning to preserve earlier knowledge and burdens resource-constrain...
OptR is proposed, an output-aware rotation method that minimizes post-W_O$ attention-output error and applies an attention-equivalent key reparameterization to reduce large channel-wise offsets without changing the softmax distribution.