This work proposes ScaleSense, a proactive, query-level resource scaling framework that addresses the critical performance-cost trade-off while maintaining low-overhead inference latency, and confirms its practical performance in production deployments.
Abstract
Cloud-native serverless data warehouses achieve fine-grained elasticity by decoupling storage from compute, yet determining the optimal resource allocation for highly heterogeneous ad-hoc queries remains a formidable industrial challenge. Our analysis of production workloads in Alibaba AnalyticDB exposes a costly ``provisioning trap'': the fear of catastrophic resource depletion drives users to blindly over-provision resources, wasting immense monetary budgets without alleviating non-CPU bottlenecks (e.g., I/O saturation). To break this impasse, we propose ScaleSense, a proactive, query-level resource scaling framework. Specifically, it features a multi-faceted query encoder that jointly models plan topologies and hardware specifications. Crucially, a quantile-based resource predictor estimates multi-dimensional physical footprints, acting as a reliable safety net for optimal resource scaling. An auto-scaling controller then navigates the performance-cost Pareto frontier, dynamically tailoring allocations to specific business priorities without requiring model retraining. Evaluations on over 1.36 million production queries show that ScaleSense achieves state-of-the-art prediction accuracy with good prediction interval coverage. By achieving a 76.7% relative improvement in optimal resource configuration selection over the best baseline, this approach addresses the critical performance-cost trade-off while maintaining low-overhead inference latency, confirming its practical performance in production deployments. Under the performance-optimization policy, ScaleSense satisfies user-defined performance requirements while reducing monetary cost by up to 5.22x.
CELLServe formalizes SLO-constrained joint resource provisioning as an optimization problem with a dedicated algorithm, and introduces an opportunistic instance merging strategy for decode phase functions to reclaim fragmented resources.
Zejian Wang, Nan Lin, Zinuo Cai et al.· ACM Transactions on Architec...· 0 citations
PRISM, a prediction-guided runtime framework that jointly selects model variants and CPU allocations for containerized edge microservices, and adapts each pipeline stage in place and minimizes predicted CPU-package energy under deadline, resource, and offline model-level Quality of Result constraints is presented.
Uwe Gropengießer, Thomas Reuter, Dominik Schön et al.· 0 citations
Y-Map is presented, a hybrid neural--symbolic framework for performance-aware ORM schema design that synthesizes valid schema candidates and predicts their performance without requiring workload execution at inference time.
Serverless computing has emerged as an attractive deployment model for deep learning model inference workflows, enabling elastic scaling and fine-grained resource billing across function instances. However, scheduling in this setting introduces a competing-objective challenge: placement decisions simultaneously govern data transfer overhead, determined by whether dependent function instances are co-located, and model loading overhead, determined by whether required model weights are memory-resident on the target node. We present AnchorDL, a joint-cost look-ahead scheduler that minimizes the combined cost of both overheads at each placement decision, with a forward term that avoids greedy suboptimality across adjacent data dependencies. Evaluated against three baselines across chain, fan-in, and fan-out workflow topologies under trace-based workloads, AnchorDL achieves the lowest median end-to-end latency across all evaluated workflows and reduces P90 latency by up to 57.4% against the model-centric baseline in the fan-out workflow. The look-ahead term further contributes substantially beyond greedy joint-cost placement.
Min Chang Kim, Hyungjun Kim, Hokun Park et al.· IEEE International Conferenc...· 0 citations
This paper employs five lightweight machine learning models to predict query-specific latency, accuracy, resource usage, and response size for each model-size-quantization-device combination, and design a lightweight heuristic that approximates the BILP solution.
Reza Farahani, Zoha Azimi, Mario Colosi et al.· 0 citations
The rapid deployment of machine learning systems across cloud, edge, and enterprise environments has brought model optimization to the forefront of systems-engineering. Despite a rich literature spanning quantization, pruning, knowledge distillation, parameter-efficient fine-tuning (PEFT), and inference-time optimization, practitioners are often left navigating these techniques through heuristics rather than principled methodology. We argue that optimization should be formulated as a constraint-driven, multi-objective engineering decision and introduce a unified framework that characterizes any production deployment along five interacting constraint dimensions: data availability, latency budget, memory budget, accuracy tolerance, and retraining budget. Building on this taxonomy, we synthesize empirical gains reported across the research literature and map them to operational constraints rather than algorithmic categories. To ensure practical relevance, we selected these techniques by reviewing recent literature for methods that report measurable improvements against critical deployment bottlenecks. We propose a prescriptive decision framework and provide optimization pipelines for four representative industrial scenarios to illustrate it in practice. To the best of our knowledge, this work provides one of the first structured attempts to formalize model optimization as a constraint-aware, multi-objective engineering process, synthesizing quantitative evidence from the research literature.
Dhruv Shivkant, S. Mohanty, S. Rai et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
