Skip to content
#federated learning Open access

FeS: Federated NLP in Few-Shot Scenarios

Dec 2026 · IEEE Transactions on Mobile Computing · Vol 25, pp. 22520-22535 · 0 citations · 88 references

TL;DR

This work is the first to explore federated NLP in the few-shot scenario (FedFSL), and introduces a system, FeS, which addresses high execution costs through three key innovations: Curriculum pacing, which injects pseudo labels into the training process in alignment with the learning progress, and Representational diversity, a strategy that selects the most learnable data for pseudo-label generation.

Abstract

The rapid growth of mobile applications (particularly deployed natural language processing applications), generate a proliferation of user-generated data, encapsulating a wealth of valuable knowledge. To facilitate the analysis and extraction of insights from on-device data while adhering to stringent data privacy regulations, natural language processing (NLP) models deployed on devices are frequently fine-tuned within a federated learning (FL) setting. However, this process typically requires hundreds of thousands of labeled training samples from mobile users. This presents a significant challenge, as users often lack the motivation or expertise to annotate their data. The scarcity of labeled data, referred to as the few-shot scenario, poses a substantial obstacle for mobile NLP applications. This work is the first to explore federated NLP in the few-shot scenario (FedFSL). Extensive measurements are conducted to highlight that previous federated NLP suffer from significant accuracy degradation when labeled data is scarce and imbalanced distributed. By leveraging advancements in pseudo-labeling and prompt learning, we establish a training pipeline that can achieve competitive accuracy on the evaluated few-shot NLP benchmarks with as few as 0.05% of training samples (fewer than 100) labeled, while the rest remain unlabeled. To make this pipeline into practice, we introduce a system, FeS, which addresses high execution costs through three key innovations: (1) Curriculum pacing, which injects pseudo labels into the training process in alignment with the learning progress; (2) Representational diversity, a strategy that selects the most learnable data for pseudo-label generation; (3) Co-planning of the model’s training depth and layer capacity. Together, these designs reduce training latency and client energy consumption by up to 46.0× and 41.2×, respectively. For network traffic, FeS achieves an average reduction of 1841.7× over FedFSL and 2.17× over the stronger FedFSL-BIAS baseline across four tasks. By integrating algorithmic and system-level innovations, FeS demonstrates that FL can be effectively applied in challenging scenarios where the majority of training samples are unlabeled.

Read PDF

Similar papers

Book Open access Nov 2026

FedPivot: Bridging Heterogeneous Multimodal Pre-training via Federated Post-Alignment

The rapid evolution of Multimodal Large Language Models (MLLMs) is bottlenecked by the saturation of high-quality public data, while vast amounts of heterogeneous multimodal data remain inaccessible in privacy-sensitive silos. Federated Learning (FL) offers a promising solution to leverage these distributed resources w...

Bao-Chen Xiong, Yifan Xu, Y. Song et al. · 0 citations
#machine learning Preprint Nov 2026

Harnessing Vision-Language Models for Federated Unsupervised Visual Recognition

Federated learning (FL) is a promising research paradigm that enables collaborative model training across distributed clients without data sharing. To handle unlabeled data in local clients, existing unsupervised FL research has primarily focused on tasks like clustering and self-supervised representation learning, lea...

Jian Liang, Kuang-Pu Guo, Ran He et al. · 0 citations
#federated learning Preprint Oct 2026

FedSSMCoOp: SSM Encoders for light-weight Federated Prompt Learning for Few-shot Classification

Vision-Language Models (VLMs) have shown strong performance across a wide range of downstream vision tasks, thanks to the complementary information contained in the respective domains. Despite the performance gains, most of these approaches rely on aligning these domains using the cosine similarity metric, which fails...

Ankita Das, Ambarish Parthasarathy, S. Channappayya et al. · 0 citations
#machine learning Review Oct 2026

FederatedBytes: Tokenizer-Free, Byte-Level Small Language Models for Efficient Multimodal Federated Learning

Federated Learning (FL) is a powerful machine learning framework that allows building a global model from the contributions of multiple, distributed clients, while ensuring the privacy of their local data. However, its heavy computational and communication overheads challenge its use for contemporary AI workloads such...

Ali Sarabadani, Massimo Piccardi, M. Abolhasan et al. · 0 citations
Open access Oct 2026

Personalized Federated Fine-Tuning of Language Models

Language Models have recently been successfully applied to a wide range of applications, typically fine-tuned using high-quality task-specific data. However, the limited accessibility of such data, due to its distribution across multiple sources and privacy constraints, makes traditional centralized training impractica...

Gabriel U. Talasso, L. Villas, A. D. de Souza · 0 citations
#machine learning Preprint Nov 2026

FedLoVA: Value-Only Aggregation for Federated LoRA Fine-Tuning of Large Language Models

Low-Rank Adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method for fine-tuning of large language models (LLMs), especially in federated learning, due to its strong performance and communication efficiency. In practice, LoRA is applied to the query and value projections of transformer layers witho...

Ensieh Khazaei, Baturalp Buyukates, Dimitrios Hatzinakos · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.