Sep 2026· IEEE Transactions on Mobile Computing· Vol 25, pp. 15221-15233· 1 citation· 43 references
Abstract
Advancements in edge computing and container technology have made it increasingly popular and convenient to deploy Large Language Models (LLMs) through containers at the edge. However, the limited GPU resources of edge servers make it impractical to retain the model in GPU memory for long periods due to the high memory cost, especially when they remain idle without user requests. Existing work unloads the entire idle models to reduce memory costs on edge servers, but reloading them introduces significant loading delays that affect task Quality of Service (QoS). Therefore, efficient management of idle models is a critical issue that has been largely neglected in existing research and requires urgent attention. To address this gap, this paper studies the problem of idle model management from the perspective of the trade-off between memory cost and loading delay under the QoS constraint. A novel layer-granularity model unloading method is proposed, which leverages the layered characteristics of the model. We formulate an online joint optimization problem to determine which layers to unload and when, and present a layer-granularity unloading strategy inspired by the ski rental problem to solve it. We implement a real system with layer-granularity unloading for LLMs on NVIDIA GPUs and validate the effectiveness of the proposed method. Experimental results show it effectively trades off memory cost and loading delay, improving overall performance by up to 39.6%.
CELLServe formalizes SLO-constrained joint resource provisioning as an optimization problem with a dedicated algorithm, and introduces an opportunistic instance merging strategy for decode phase functions to reclaim fragmented resources.
Zejian Wang, Nan Lin, Zinuo Cai et al.· ACM Transactions on Architec...· 0 citations
A comprehensive review of Kubernetes scheduling strategies published between January 2023 and January 2026 is presented and a multi-dimensional taxonomy is established that categorizes scheduling approaches based on common objectives, modification methods, optimization methodologies, targeted workloads, evaluation methods, scheduling scopes, and performance metrics.
Mohammed Alhakimi, R. Latip· Computers· 0 citations
Festina is presented, a profiling-guided, power-aware control plane to minimize cluster-wide energy for serverless LLM serving and performs energy-aware workload consolidation to reduce GPUs'static power consumption via SLO-aware migration.
Tianyu Wang, Gourav Rattihalli, A. Dhakal et al.· arXiv.org· 0 citations
The new EMC+ proposal is an OS‐driven elasticity manager for container‐based environments that continuously estimates idle core cycles left by regular (inelastic) applications, and reallocates idle cores to elastic ones, even during short time intervals, and has minimal impact on the performance and QoS of colocated inelastic applications.
J. C. Saez, Carlos Bilbao, Manuel Prieto-Matías· Concurrency and Computation· 0 citations
Large language model (LLM) deployment at the network edge faces a fundamental paradox: applications require full-scale models for sophisticated reasoning, yet edge devices impose severe resource constraints across computation, memory, and network. Existing approaches fail to effectively orchestrate resources across the edge-cloud continuum, leaving capacity underutilized while struggling with heterogeneous and volatile distributed environments. We present DynoPipe, an adaptive edge-cloud system that addresses these constraints through dynamic pipeline parallelism with shifting computational boundaries. DynoPipe tackles three core challenges: structural heterogeneity causing 94% pipeline idle time, temporal resource volatility invalidating static partitioning, and boundary migration overhead trapping systems in suboptimal configurations. Through boundary-constrained pipeline construction, proactive multi-configuration orchestration, and hierarchical state management, DynoPipe eliminates the memory wall while preserving data locality, achieving $\mathbf{1 0. 1} \times$ throughput improvement over edge-only baselines and $\mathbf{1. 6} \times$ over cloud-only execution, with 99.2% latency reduction.
Yanying Lin, Baicheng Chen, Xinyu Zhang et al.· International Symposium on C...· 0 citations
Known for his clear and elegant writing style, Bertsekas shaped fields from control and optimization to large-scale computation and artificial intelligence.
