Yuxiong He

Heterogeneous SLO Guaranteed Multi-Resource-Aware Batching in LLM Serving

In this paper, we study a mixed-prompt scenario—where both short and long prompts coexist—in an LLM inference serving system that supports diverse applications with heterogeneous iteration-time SLOs. To improve throughput for long prompts, prior work divides them into chunks and batches requests or chunks to meet the t...