End-to-End Latency-Minimizing and Load-Balanced Request Scheduling for Edge LLM Inference in Agentic AI Services
An online request scheduling framework for edge LLM inference that jointly minimizes long-term average end-to-end latency and regulates workload distribution across heterogeneous edge servers is investigated and the LYREO approach, a cross-slot inference model that captures transmission, prefill, iteration-level decodi...
Zhen Li, Jun Cai, Hao-Ran Gao et al.
· 0 citations