Skip to content
Open access

Personalized Federated Fine-Tuning of Language Models

Oct 2026 · Anais do XV Concurso de Teses e Dissertações em Inteligência Artificial e Computacional (CTDIAC 2026) · 0 citations · 38 references

Abstract

Language Models have recently been successfully applied to a wide range of applications, typically fine-tuned using high-quality task-specific data. However, the limited accessibility of such data, due to its distribution across multiple sources and privacy constraints, makes traditional centralized training impractical. Federated Learning emerges as an alternative by enabling distributed model training without sharing raw data. Nevertheless, federated adaptation introduces two main challenges: the high training and communication costs and the heterogeneity of data, which differ in domains and distributions. To address these challenges, this dissertation develops personalized federated adaptation methods that combine client clustering with parameter-efficient fine-tuning techniques, resulting in four main contributions: (i) a model-similarity-based client clustering method for classical FL scenarios; (ii) a parameter-efficient fine-tuning pipeline for federated language models and its trade-offs; (iii) a method that integrates client clustering with efficient adaptation to build multilingual language models specialized for each language; and (iv) a novel task-centric personalization approach that combines local and global clustering mechanisms with adaptive inference-time routing. The proposed methods improved both performance and training efficiency compared with several state-of-the-art approaches under resource-constrained and heterogeneous data settings, demonstrating their effectiveness and practical relevance.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.