AI Networking Cookbook: Practical recipes for AI-assisted network automation and development
CELLServe formalizes SLO-constrained joint resource provisioning as an optimization problem with a dedicated algorithm, and introduces an opportunistic instance merging strategy for decode phase functions to reclaim fragmented resources.
Festina is presented, a profiling-guided, power-aware control plane to minimize cluster-wide energy for serverless LLM serving and performs energy-aware workload consolidation to reduce GPUs'static power consumption via SLO-aware migration.
Frontier open-weight models are increasingly available, but serving them still largely assumes datacenter infrastructure. We present FreeToken, an edge-native MoE serving system that treats a personal machine not as a small GPU, but as a unified, elastic inference platform. FreeToken co-designs the full serving stack,...
A ReRAM near-memory architecture that keeps expert weights resident behind high-bandwidth local reads and recovers occupancy with bounded core-local multicast pooling, coactivation-aware placement, and load-aware fetch, and sizes each communication level from induced demand is presented.
Roomie is presented, a model serving orchestration architecture that predicts and avoids kernel-level interference between colocated DNNs, and reduces SLO violations by up to 3x, while maintaining comparable, and in many cases superior, goodput relative to existing approaches.
This paper employs five lightweight machine learning models to predict query-specific latency, accuracy, resource usage, and response size for each model-size-quantization-device combination, and design a lightweight heuristic that approximates the BILP solution.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.