Serverless computing has emerged as an attractive deployment model for deep learning model inference workflows, enabling elastic scaling and fine-grained resource billing across function instances. However, scheduling in this setting introduces a competing-objective challenge: placement decisions simultaneously govern data transfer overhead, determined by whether dependent function instances are co-located, and model loading overhead, determined by whether required model weights are memory-resident on the target node. We present AnchorDL, a joint-cost look-ahead scheduler that minimizes the combined cost of both overheads at each placement decision, with a forward term that avoids greedy suboptimality across adjacent data dependencies. Evaluated against three baselines across chain, fan-in, and fan-out workflow topologies under trace-based workloads, AnchorDL achieves the lowest median end-to-end latency across all evaluated workflows and reduces P90 latency by up to 57.4% against the model-centric baseline in the fan-out workflow. The look-ahead term further contributes substantially beyond greedy joint-cost placement.
Min Chang Kim, Hyungjun Kim, Hokun Park et al.· IEEE International Conferenc...· 0 citations
Large Language Model (LLM) inference services increasingly rely on heterogeneous GPU clusters to balance cost and performance. However, request routing in such environments is challenging because schedulers must account for hardware heterogeneity, dynamic workload characteristics, and bursty arrivals. Existing approaches either ignore hardware differences, rely on static heterogeneity-aware allocation, or use queue-based proxies that fail to capture the true remaining work under load. We present BYSTANDER, a prediction-based scheduling framework that uses a small language model (SLM) to estimate End-to-End (E2E) latency of request for each GPU pool from request features and current pool state. BYSTANDER then adaptively narrows the candidate pools via Fisher–Jenks grouping and performs queue-aware selection within that set. Its pool-based design keeps prediction overhead scalable as cluster size grows. We evaluate BYSTANDER on ShareGPT and LMSYS-Chat workloads over heterogeneous clusters of 3–7 GPUs (RTX 3090/4090/5090). Compared with round-robin, oracle-weighted round-robin, shortest-queue-first, and SLM Adaptive baselines, BYSTANDER reduces P99 E2E latency by up to 63.4% and P99 time-to-first-token by up to 87.4% under bursty traffic.
Minjae Jung, Hyungjun Kim, Hokun Park et al.· IEEE International Conferenc...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
