AccSI: Accelerating Serverless Inference via Edge-Assisted Pipelined Model Fetching

Abstract

Serverless edge computing (SEC) is an emerging paradigm for delivering low-latency and auto-scaling services on resource-limited edges. However, when applied to DNN inference applications, SEC suffers from significant cold-start overhead, because large model parameters need to be fetched from remote model registries before execution. Prior studies reduce model-fetching latency by prefetching models in SSDs. However, these methods are impractical for SEC, as resource-constrained edge servers cannot accommodate all required DNN models. To solve it, we propose AccSI, a novel edge-assisted pipelined model fetching framework to accelerate serverless inference. AccSI aggregates distributed edge storage to prefetch DNN model layers and employs a pipelined execution strategy that overlaps model fetching, model loading, and inference, thereby minimizing the overall application completion time (ACT). To fully unlock AccSI’s potential, we jointly optimize application placement and model fetching decisions with a provable approximation ratio. In addition, considering the dynamic nature of SEC environments and application requests, we adaptively adjust application placement, model fetching, and prefetching decisions at runtime. Finally, extensive experiments demonstrate that AccSI achieves a speedup of up to <inline-formula><tex-math notation="LaTeX">$3.47\times$</tex-math><alternatives><mml:math><mml:mrow><mml:mn>3</mml:mn><mml:mo>.</mml:mo><mml:mn>47</mml:mn><mml:mo>×</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="dong-ieq1-3709190.gif"/></alternatives></inline-formula> in average ACT.