A decision-oriented framework that translates theoretical insights into practical guidelines is introduced, establishing a structured foundation for broader communities to train and deploy human-centered AI systems sustainably and efficiently.
Abstract
The rapid advancement of artificial intelligence (AI), particularly large language models (LLMs), has created a significant socio-technical divide. The immense computational resources required for AI training increasingly limit participation to a few well-funded entities, hindering the democratization of AI research and raising concerns about environmental sustainability. While optimization algorithms are critical to reducing these resource barriers, the current landscape is highly fragmented, offering limited practical guidance for practitioners in resource-constrained environments. To address this accessibility gap, we present a unified taxonomy of deep learning optimizers that systematically organizes methods by their order of information: zeroth, first, and second order, while integrating emerging, IO-aware and Flash attention paradigms. Instead of merely enumerating algorithms, our approach emphasizes cost-efficiency, memory usage, and hardware constraints as pivotal factors for equitable AI development. Our synthesis of the literature reveals that system-level considerations, particularly IO efficiency, are essential not just for computational performance, but for making large-scale AI accessible. We introduce a decision-oriented framework that translates theoretical insights into practical guidelines, establishing a structured foundation for broader communities to train and deploy human-centered AI systems sustainably and efficiently.
The increasing intricacy and scale of deep learning (DL) models have heightened the necessity for effective weight and parameter optimization algorithms that sustain superior performance while reducing computational resource consumption. This paper thoroughly analyzes the evolution of parameter optimization strategies, ranging from initial methodologies to modern advancements, elucidating their principles and applications in natural language processing (NLP) and machine learning (ML). We pay special emphasis to parameter-efficient fine-tuning (PEFT) approaches, such as low-rank adaptation (LoRA) and its extensions, which facilitate the adaptation of large language models (LLMs) on resource-limited devices. These methods address challenges such as high computational demands, energy consumption, and deployment restrictions, thereby promoting more accessible and environmentally sustainable artificial intelligence (AI) solutions. By integrating methodological understandings with recent advancements, this survey underscores the essential role of parameter optimization in enabling scalable deep learning systems. It serves as an essential guide for researchers seeking to apply these techniques across diverse domains, emphasizing their impact on achieving efficient and robust model performance.
M. Najafi, Ehsan Tavan, Stefan W. Kuhn et al.· Artificial Intelligence Revi...· 0 citations
This dataset defines a new basis for reproducible and data-driven AI design, advancing the emerging paradigm of LLM-driven AutoML and architectural generalization across modalities and hardware.
Tolgay Atinc Uzun, Waleed Khalid, Saif U Din et al.· 19 citations
Generative Artificial Intelligence (GenAI) leverages generative models, including Diffusion models and Foundation Models (FMs) such as Large Language Models (LLMs), to generate new data. GenAI has attracted growing attention as it enables applications such as text generation, image synthesis, and multimodal reasoning. However, deploying such GenAI models on resource-constrained edge devices poses key challenges. Edge devices typically have limited computational power and strict energy budgets. They also face the memory-wall problem, where the cost of moving data between memory and compute units exceeds the cost of computation itself. This survey provides a comprehensive overview of techniques that enable GenAI deployment at the edge, covering software optimizations, hardware innovations, and system-level frameworks. While all three categories are discussed, particular emphasis is given to hardware-focused approaches. We review recent hardware studies and classify them into three main levels: computation, memory, and scheduling, with eight finer-grained subcategories that capture the mainstream hardware techniques. Many of these are hardware–software co-design strategies that align GenAI workloads with edge resource constraints. By synthesizing state-of-the-art solutions and identifying open challenges, this survey outlines a roadmap for building efficient GenAI systems at the edge.
Mozhgan Navardi, Yuzhe Fu, Yueqian Lin et al.· ACM Computing Surveys· 0 citations
The design of transformer-based Large Language Models (LLMs) is being radically changed through new architectures that are able to overcome scalability limitations of previous designs, including Mixture-of-Experts (MoE), Multi-Head Latent Attention (MLA), and Multi-Token Prediction (MTP). As an open-weighted model released at the end of 2024, which has both state of the art architectural transparency and production scale efficiency, DeepSeeek-V3 represents the ultimate testing ground for investigating these modern technologies. This paper provides a comprehensive analysis of the architectural structure of DeepSeek-V3 based upon information from the DeepSeek-V3 Technical Report, industry benchmarking data and independent latency testing, to demonstrate how various techniques can be used to optimize training while still providing competitive performance in code generation and mathematical reasoning. In addition, latency testing conducted on a Distilled version of DeepSeek-V3, with approximately 14 billion parameters, running on a T4 GPU, reveals that although significant improvements have been made in optimizing latency there remains substantial barriers to deploying these models. Through this context, this research will serve as a reference document for practitioners and researchers who wish to understand current trends and challenges in increasing accessibility to high performance AI models.
Yassine Zouhdi, B. Hdioud· EPJ Web of Conferences· 0 citations
With the growing deployment of state-of-the-art deep neural networks in safety-critical, embedded and edge-computing applications, there is a strong incentive to design models that achieve high accuracy while maintaining very limited computational and memory budgets. We present a systematic study in resource-efficient AI model design addressing two orthogonal strategies of structured parameter reduction and accuracy-aware compression. Based on experiments on 6 benchmark datasets Image Net, CIFAR-10, GLUE (SST-2 and MNLI), MS COCO and Squad 1.1 we evaluate and compare pruning, quantization-aware training (QAT), knowledge distillation (KD), low-rank factorization and neural architecture search (NAS) in a systematic manner. The proposed hybrid pipeline includes structured pruning, INT8 quantization and task-specific knowledge distillation, which is benchmarked against standalone methods. Empirical results show that the proposed hybrid gives 4.5–5.2× inference speedup, 6–8× parameter reduction but just −0.2 to −0.3 percentage points accuracy drop compared to full-precision baselines on vision and language tasks. Five contextual analytical tables, capturing performance across the parameters latency alone, energy consumption alone and cross-task accuracy reinforce that Pareto-optimal results are consistently achieved for this hybrid method. Comparison with fundamental earlier research including Han et al. [5], Hinton et al. [12], Jacob et al. [9], Hu et al. [16], and Sanh et al. Now, looking at [20], it reinforces the idea of upper bound projection based approach for accuracy-oriented, multi-level compression. These results have immediate application to large-scale AI deployment on resource-limited hardware platforms, allowing AI democratization with fidelity.
Krishna Kumar Tiwari, Komal Tahiliani, Uma Shankar Birthare et al.· International journal of com...· 0 citations
Multimodal large language models (MLLMs) have made it possible for artificial-intelligence systems to reason jointly across vision and language, supporting tasks ranging from image captioning and visual question answering to clinical decision support and autonomous perception. As MLLM scale grows, however, deploying these models in cost- and energy-bounded cloud environments has become a defining engineering challenge. This mini-review consolidates recent literature at the intersection of four research strands: (i) multimodal architecture design and fusion strategies, (ii) neural-activation patterns and mechanistic specialization, (iii) selective and conditional computation including mixture-of-experts, and (iv) cloud-deployment optimisation. We propose a unified four-quadrant taxonomy of efficiency strategies, trace the field's evolution through a decade-scale timeline, and synthesise sixteen primary studies in cross-cutting comparison tables. Particular attention is given to the under-explored intersection of domain adaptation and efficiency, where evidence is converging that neuron-level domain awareness can simultaneously reduce inference cost and improve interpretability. We identify five persistent limitations of current methods and five concrete research gaps that follow from them. The review closes with an integrated future-research agenda built on three converging innovations: adaptive cross-modal attention re-weighting, knowledge-injection pathways, and sparse domain-conditioned neuron gating, and outlines the cloud-aware evaluation framework that would validate them. The article is intended as a single-source reference for researchers and practitioners designing efficient, trustworthy multimodal AI for cloud deployment
Olom Ogar Austin, Joshua Abah, Ali Muhammad et al.· International journal of com...· 0 citations