Accelerating Federated LoRA Fine-Tuning for LLMs with Adaptive Ranks at Network Edge
TL;DR
FLFAR reduces federated LoRA fine-tuning time by up to 40% while maintaining comparable accuracy, and at the beginning of each global round, the LoRA rank of each device is adapted according to its computational and communication capacity.
Abstract
The rapid development of large language models (LLMs) has played a crucial role in advancing artificial intelligence. Pretrained LLMs can be adapted to various downstream tasks through fine-tuning. To alleviate the intensive resource requirements of fine-tuning and safeguard data privacy, researchers have integrated Low-Rank Adaptation (LoRA) with Federated Learning (FL), giving rise to a new framework known as Federated LoRA fine-tuning. However, due to heterogeneous computation and communication capacities across devices, federated LoRA fine-tuning still suffers from the straggler problem when using fixed LoRA ranks. In this work, we propose a method named FLFAR, which accelerates federated LoRA fine-tuning with adaptive ranks. Specifically, at the beginning of each global round, the LoRA rank of each device is adapted according to its computational and communication capacity. As a result, devices can complete fine-tuning nearly simultaneously, effectively reducing synchronization delays. Extensive experiments show that, compared with baseline methods, FLFAR reduces federated LoRA fine-tuning time by up to 40% while maintaining comparable accuracy.