A Survey of Inference Processing Units for Large Language Model Inference: Chips, Systems, Algorithms, and Paradigms (2024–2026)

Abstract

This survey reviews 2024–2026 progress in LLM inference through a unified "inference processing unit" (IPU) lens, organized into 4 coupled layers. 1. The chip layer covers dedicated accelerators (Groq's LPU, Cerebras' wafer-scale engine), hyperscaler ASICs (Google's TPU v7, AWS Trainium 3, Microsoft's Maia 200), and emerging vendors in China, South Korea, and Europe, with MLPerf results. 2. The system layer analyzes paged attention, continuous batching, and prefill–decode disaggregation in vLLM, SGLang, and TensorRT-LLM. 3. The algorithm layer reviews KV cache compression, low-bit quantization, and speculative decoding. 4. The paradigm layer examines test-time compute scaling, reasoning models, and multi-agent and agentic inference. Across all four layers we distill three recurring patterns of system evolution (Section 7.6): 1. what becomes routine moves into hardware, 2. cheap prediction invites more prediction, 3. each workload is routed to the cheapest adequate substrate. We close with open problems—the memory wall, energy and cost, long context, evaluation, ecosystem fragmentation, security, and scheduling—and future directions: hardware convergence, an emerging "inference resource management" layer, inference as a computational primitive, regional ecosystems, and edge–cloud collaboration.