Book
Open access
Jul 2026
PKAS: Predictive KVCache-Aware Scheduling for Faster LLM and Transformer Inferences
With rising popularity of LLMs, the performance, scalability, and resource-efficiency of inferences become a crucial challenge. The core part of the inference process is the KV cache, which avoids recomputing intermediate attention states, and the batching strategy that batches multiple requests per forward pass to lev...
Jie Ye, Avinash Maurya, Krishna Teja Chitty-Venkata et al.
· IEEE International Symposium... · 1 citation