Mix-or-Split: Latency-Aware Scheduling for Edge–Cloud LLM Inference

Abstract

In this paper, we present a latency-aware scheduler for large-language-model (LLM) inference across mobile devices, edge servers, and a remote cloud. Our fine-grained delay model captures OFDMA uplink/downlink rates, KV-cache backhaul serialization, and profiled GPU planning-chunk resource constraints, enabling per-request routing among (i) cloud-only execution, (ii) cloud-prefill + edge-decode split execution, and (iii) edge prefill–decode mix execution. The scheduler targets high goodput and SLO attainment under hard deadlines, TTFT/TPOT targets, and guarded SM-resource constraints via a four-stage loop: windowed admission, $\delta$δ-similar prompt bucketing, tile selection, and SM-feasible prefill/decode allocation with dynamic execution adjustment. We bound padding waste, prefill-latency standard deviation, and the number of buckets, and analyze how the feasible allocation frontier is shaped by active register, SMEM, and warp walls under high-bandwidth, low-load conditions. On an A800/A6000 heterogeneous testbed, repeated real-serving experiments exercise all three routing decisions. In a Qwen3-8B heterogeneous replay, the online policy lowers mean end-to-end latency by 6.4% relative to the best fixed mean baseline and attains 100% SLO, while its P95 confidence interval overlaps those of fixed Cloud and edge-local execution. In a heterogeneous Llama-3 replay, the policy selects Cloud/Mix/Split for 18/21/6 requests; it attains 100% SLO across the four primary workload regimes and 95.6--100% across the offered-load points. Real CUDA-launch analysis further shows that the dominant Prefill/Decode kernel pair fails the joint-residency resource bound for all 37 Llama shapes on both GPUs; guarded alternative pairs account for 41.4% and 26.9% of the analyzed pair weight on A800 and A6000, respectively.