InfiniLoRA: Disaggregated Multi-LoRA Serving for Large Language Models
LoRA enables efficient customization of LLMs and is widely used in multi-tenant and multi-task serving. However, emerging model architectures such as MoE significantly increase LoRA memory cost, making existing coupled LoRA serving designs poorly scalable and prone to tail-latency inflation. We present InfiniLoRA, a di...
Hongyu Chen, Letian Ruan, Zilin Xu et al.
· Workshop Proceedings of the... · 0 citations