Serverless edge computing, despite its flexibility and efficiency, is hindered by high startup latency during peak load. Remote fork, employing either Checkpoint/Restore (C/R) or Remote Direct Memory Access (RDMA), offers a potential solution for function scaling acceleration. Although RDMA fork is faster, the opportunities are limited, whereas C/R fork is more common but slower. Moreover, the regeneration capability that a forked function can further fork new instances complicates the remote fork decisions for fast scaling. Therefore, in this paper, we are motivated to address the problem on how to holistically exploit C/R fork and RDMA fork with the consideration of the underlying infrastructure features (e.g., topology, resources, etc.) to realize fast function scaling. We first formulate it to an Integer Linear Programming (ILP) problem. We further introduce a Heat metric to assess the potential of an edge server as a fork destination according to the topology and resource availability, and propose a Heat-based fork strategy (HEAT) for both the fork destination site and the corresponding fork mode decisions. Experiment results demonstrate that HEAT improves the function scaling speed by 46% and RDMA resource utilization by 51%, compared to state-of-the-art scaling solutions.
Zhe-Xiong Li, Deze Zeng, Lin Gu et al.· IEEE Transactions on Mobile...· 0 citations
Modern large language model (LLM) inference suffers from severe Time-To-First-Token (TTFT) bottlenecks. Existing prefix KV caching mechanisms are inherently stateless, forcing a trade-off between cross-chunk attention accuracy and online recomputation overhead. To address this issue, we propose Pegasus, a novel stateful prefix KV caching system that aims to achieve full-context attention accuracy while avoiding costly recomputation. To handle the exponential growth of context states under limited memory capacity, Pegasus employs a Recursive Path-Pruning Caching (RPPC) algorithm to selectively cache high-value states based on access frequency, memory footprint, and asymmetric latency benefit. In addition, Pegasus introduces a transition-based KV management mechanism to mitigate cache-miss overhead. By exploiting the sparsity of state-dependent KV variations, it replaces expensive attention recomputation and I/O-intensive tensor reloading with lightweight sparse state transitions. Extensive experiments show that Pegasus improves end-to-end serving throughput by 45.9% on average, reduces TTFT by up to 78.5%, and lowers cache-miss recovery overhead by more than 72%.
Fahao Chen, Peng Li, Dongxiao Yu et al.· Fall Joint Computer Conferen...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
