Exploring High-Bandwidth Flash for Modern LLM Inference: Opportunities and Challenges
Analysis of HBF-based LLM-serving systems under diverse system configurations and operating scenarios shows that HBF can significantly improve the batch size, throughput, and flexibility of LLM-serving systems while reducing the minimum GPU requirements, but realizing these benefits critically depends on sustaining HBM...