Improving Inference Efficiency in Deep Neural Networks

Abstract

The increasing complexity of modern deep neural networks (DNNs), particularly Transformer architectures such as large language models (LLMs) and diffusion models, has made inference efficiency a central challenge across edge and datacenter platforms. The two regimes, however, impose sharply different constraints. At the edge, limited memory capacity, energy budgets, and compute resources conflict with the need for low latency under small batch sizes. In datacenters, efficiency is driven primarily by throughput and cost at scale. This thesis develops techniques to improve inference efficiency in both regimes through techniques ranging from advanced quantization and model compression to optimizing non-linear Softmax operations and batched mixture-of-expert model inference.First, we focus on quantizing diffusion models to enable visual content generation at the edge. We show that aggressive post-training quantization degrades diffusion model quality, with heterogeneous quantization offering a pathway to better trade-offs among model size, compute requirements, and quality. We propose HNMQ, a new approach to heterogeneous quantization that follows a fine-grained structured N:M pattern in which N of every M weights are kept at a lower precision. This enables hardware-friendly sub-4-bit configurations that match the quality of content generated by 4-bit models with improved latency and energy.Second, we address activation-side inefficiency in quantized LLM inference, where activations often remain at a higher precision than weights. SPARQLe decomposes higher-precision activations into dense low-order and sparse high-order components, enabling accurate execution on narrow hardware datapaths while reducing computation and data movement.Third, we analyze pruning in ultra-low-precision DNNs and show that conventional sparsity-based model compression degrades because of indexing overheads and reduced attainable sparsity. Seprox instead exploits value repetition induced by low-precision representations, improving compression efficiency while reducing inference energy and latency.Fourth, we examine Softmax, a major bottleneck in the attention computation of Transformer models: it limits parallelism and maps poorly to NPUs. Softprox is a post-fine-tuning framework that reduces Softmax complexity through attention reuse, neural network approximation, and statistical replacement of the maximum operation, enabling inference speedups on commodity hardware without retraining.Finally, we address the sublinear throughput scaling of batched MoE inference, which arises because different tokens in a batch select different experts, growing the set of expert weights that must be fetched per batch. TokenMix is a training-free technique that mixes token representations within the router network to encourage coordinated expert selection, thereby increasing expert reuse across a batch and improving throughput.Collectively, these techniques provide an effective pathway to improving the efficiency of AI inference.