DR-EFT: Exploring and reloading domain-representative experts for the memory-constrained fine-tuning of MoE large models.
An algorithm framework named DR-EFT (Domain-Representative Experts for Fine-Tuning), which explores and loads the domain-representative experts for subsequent retraining and reincorporation and demonstrates robustness through validations on popular MoE LLMs, including Qwen, DeepSeek, and Ernie.