Dec 2026· IEEE Transactions on Mobile Computing· Vol 25, pp. 22520-22535· 0 citations· 88 references
TL;DR
This work is the first to explore federated NLP in the few-shot scenario (FedFSL), and introduces a system, FeS, which addresses high execution costs through three key innovations: Curriculum pacing, which injects pseudo labels into the training process in alignment with the learning progress, and Representational diversity, a strategy that selects the most learnable data for pseudo-label generation.
Abstract
The rapid growth of mobile applications (particularly deployed natural language processing applications), generate a proliferation of user-generated data, encapsulating a wealth of valuable knowledge. To facilitate the analysis and extraction of insights from on-device data while adhering to stringent data privacy regulations, natural language processing (NLP) models deployed on devices are frequently fine-tuned within a federated learning (FL) setting. However, this process typically requires hundreds of thousands of labeled training samples from mobile users. This presents a significant challenge, as users often lack the motivation or expertise to annotate their data. The scarcity of labeled data, referred to as the few-shot scenario, poses a substantial obstacle for mobile NLP applications. This work is the first to explore federated NLP in the few-shot scenario (FedFSL). Extensive measurements are conducted to highlight that previous federated NLP suffer from significant accuracy degradation when labeled data is scarce and imbalanced distributed. By leveraging advancements in pseudo-labeling and prompt learning, we establish a training pipeline that can achieve competitive accuracy on the evaluated few-shot NLP benchmarks with as few as 0.05% of training samples (fewer than 100) labeled, while the rest remain unlabeled. To make this pipeline into practice, we introduce a system, FeS, which addresses high execution costs through three key innovations: (1) Curriculum pacing, which injects pseudo labels into the training process in alignment with the learning progress; (2) Representational diversity, a strategy that selects the most learnable data for pseudo-label generation; (3) Co-planning of the model’s training depth and layer capacity. Together, these designs reduce training latency and client energy consumption by up to 46.0× and 41.2×, respectively. For network traffic, FeS achieves an average reduction of 1841.7× over FedFSL and 2.17× over the stronger FedFSL-BIAS baseline across four tasks. By integrating algorithmic and system-level innovations, FeS demonstrates that FL can be effectively applied in challenging scenarios where the majority of training samples are unlabeled.
The rapid evolution of Multimodal Large Language Models (MLLMs) is bottlenecked by the saturation of high-quality public data, while vast amounts of heterogeneous multimodal data remain inaccessible in privacy-sensitive silos. Federated Learning (FL) offers a promising solution to leverage these distributed resources w...
Bao-Chen Xiong, Yifan Xu, Y. Song et al.· Proceedings of the 34th ACM...· 0 citations
Federated learning (FL) is a promising research paradigm that enables collaborative model training across distributed clients without data sharing. To handle unlabeled data in local clients, existing unsupervised FL research has primarily focused on tasks like clustering and self-supervised representation learning, lea...
Jian Liang, Kuang-Pu Guo, Ran He et al.· 0 citations
Vision-Language Models (VLMs) have shown strong performance across a wide range of downstream vision tasks, thanks to the complementary information contained in the respective domains. Despite the performance gains, most of these approaches rely on aligning these domains using the cosine similarity metric, which fails...
Ankita Das, Ambarish Parthasarathy, S. Channappayya et al.· 0 citations
Federated Learning (FL) is a powerful machine learning framework that allows building a global model from the contributions of multiple, distributed clients, while ensuring the privacy of their local data. However, its heavy computational and communication overheads challenge its use for contemporary AI workloads such...
Ali Sarabadani, Massimo Piccardi, M. Abolhasan et al.· 0 citations
Language Models have recently been successfully applied to a wide range of applications, typically fine-tuned using high-quality task-specific data. However, the limited accessibility of such data, due to its distribution across multiple sources and privacy constraints, makes traditional centralized training impractica...
Gabriel U. Talasso, L. Villas, A. D. de Souza· Anais do XV Concurso de Tese...· 0 citations
Low-Rank Adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method for fine-tuning of large language models (LLMs), especially in federated learning, due to its strong performance and communication efficiency. In practice, LoRA is applied to the query and value projections of transformer layers witho...
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
MIT News · Artificial Intelligence· news.mit.eduNov 17, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 9, 2026