Task-Aware Federated Fine-Tuning for MoE-based Large Language Models
FedTAR is proposed, a task-aware federated fine-tuning method for MoE-based LLMs that establishes the association between local updates and task preference via routing outputs and preserves expert specialization and mitigates destructive interference among heterogeneous clients.
Ting-Qi Wang, Hongyu Ke, Hao-Xin Wang et al.
· 0 citations