Characterizing High Bandwidth Flash for LLM Serving
Large language model (LLM) serving requires substantial memory to store model weights and KV caches. As models grow larger and contexts become longer, memory capacity and bandwidth increasingly become bottlenecks for serving performance. Agentic workloads compound this pressure through repeated interactions over growin...
Zack Yu, Chloe Wong, C. Hooper et al.
· 0 citations