A Unified Taxonomy of Deep Learning Optimizers for Scalable and Efficient AI Systems
TL;DR
A decision-oriented framework that translates theoretical insights into practical guidelines is introduced, establishing a structured foundation for broader communities to train and deploy human-centered AI systems sustainably and efficiently.
Abstract
The rapid advancement of artificial intelligence (AI), particularly large language models (LLMs), has created a significant socio-technical divide. The immense computational resources required for AI training increasingly limit participation to a few well-funded entities, hindering the democratization of AI research and raising concerns about environmental sustainability. While optimization algorithms are critical to reducing these resource barriers, the current landscape is highly fragmented, offering limited practical guidance for practitioners in resource-constrained environments. To address this accessibility gap, we present a unified taxonomy of deep learning optimizers that systematically organizes methods by their order of information: zeroth, first, and second order, while integrating emerging, IO-aware and Flash attention paradigms. Instead of merely enumerating algorithms, our approach emphasizes cost-efficiency, memory usage, and hardware constraints as pivotal factors for equitable AI development. Our synthesis of the literature reveals that system-level considerations, particularly IO efficiency, are essential not just for computational performance, but for making large-scale AI accessible. We introduce a decision-oriented framework that translates theoretical insights into practical guidelines, establishing a structured foundation for broader communities to train and deploy human-centered AI systems sustainably and efficiently.