TL;DR
SolidAttention is introduced, an LLM inference engine which addresses limitations through a tight co-design of dynamic attention sparsity algorithms and SSD-based storage management and minimizes SSD-induced blocking latency.
The challenges of the restoration-recomputation trade-off are investigated and its impact on inference performance when left unaddressed, and an I/O-aware KV-cache management policy is presented that dynamically navigates this trade-off.
DynamoServe is presented, a multi-tenant LLM serving framework that addresses challenges through three key innovations: leveraging stranded GPU memory to offload model weights and KV caches, mitigating resource fragmentation in multi-workload environments, and improving memory locality through coordinated data placemen...
This model reveals one key opportunity: dividing a restore request proportionally between the storage path and the GPU can improve inference performance while still meeting SLOs, and reduces the KV-cache storage stack to a performance model based on per-tier capacity, per-tier and interconnect bandwidth, and GPU arithm...
HARE introduces a holistic allocation model that captures the demand correlation between cache size and other resources, and uses a novel two-phase harvest/redistribute method to optimize resource allocation across tenants, maximizing the throughput of each while maintaining fairness.
The results establish bounded feasibility and identify limitations that a deployment claim must not hide, as well as characterize cache-policy sensitivity, numerical comparison boundaries, and measurement limits.
Log-structured merge-tree (LSM-tree) key-value stores rely on caching to mitigate multi-component lookups and long tails, yet block and KP caches are prone to compactioninduced expiry, and all three cache types suffer from scan pollution under LRU eviction. We present AutoThermKV, a two-level in-memory architecture tha...
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
