AOS-R (Adaptive Optimizer Switching, Rule-Based), a lightweight controller that monitors six online gradient-space signals and switches among AdamW, SGD-M, and Lion as the optimization landscape evolves, and achieves best accuracy on 6 of 8 combinations with a mean +0.4 pp gain.
Abstract
Single-optimizer training is a poor fit for the distinct phases of deep network optimization: adaptive methods handle noisy early gradients well but overshoot flat minima, while SGD with momentum generalizes better in the late phase but converges slowly early on. We introduce AOS-R (Adaptive Optimizer Switching, Rule-Based), a lightweight controller that monitors six online gradient-space signals -- gradient noise scale (GNS), Hutchinson curvature trace, loss stagnation, update stability ratio, gradient stability index (GSI), and loss improvement ratio (LIR) -- and switches among AdamW, SGD-M, and Lion as the optimization landscape evolves. State-preserving momentum transfer and a 400-step learning-rate bridge prevent accuracy degradation at every transition point. On CIFAR-100/WRN-28x10, AOS-R reaches 78% top-1 in 81 epochs -- 26% fewer than AdamW (109), 43% fewer than SGD-M (143), and 16% fewer than Lion (96). Across eight model-dataset benchmarks, AOS-R achieves best accuracy on 6 of 8 combinations with a mean +0.4 pp gain and 0.80x convergence speedup over AdamW under a single shared hyperparameter configuration.
Deep neural networks often encounter non-convex optimization challenges during training due to the presence of local minima, saddle points, and complex loss surfaces. Existing optimization algorithms such as Adam and Stochastic Gradient Descent (SGD) offer complementary advantages—Adam provides faster convergence, while SGD tends to achieve better generalization. However, neither optimizer alone effectively balances both properties in non-convex settings. To address this limitation, this paper proposes a phase-switch hybrid optimization strategy that combines the strengths of Adam and SGD. The proposed method employs Adam during the initial phase of training to enable rapid convergence and efficient exploration of the loss landscape, and then switches to momentum-based SGD in the later phase to improve generalization and ensure stable convergence. The effectiveness of the proposed approach is evaluated on three benchmark dataset the M4 time-series forecasting data set, under different learning rate settings. Experimental results demonstrate that the proposed method achieves performance that is superior or comparable to existing optimizers in terms of accuracy and loss minimization. These results indicate that the proposed hybrid optimization strategy provides a simple and effective solution for handling non-convex optimization problems
Optimization plays a critical role in training deep neural networks, directly impacting convergence speed, model generalization, and stability. While existing methods such as stochastic gradient descent (SGD) and adaptive optimizers like Adam and AdamW have achieved significant success, they exhibit limitations in handling extreme gradients and noisy updates, and maintaining stable convergence across diverse architectures and datasets. In this study, we propose Quartz, a novel optimizer that combines momentum accumulation with bounded adaptive scaling to improve convergence efficiency and robustness. Quartz introduces a gradient-saturation mechanism that prevents excessively large updates and enforces adaptive step size bounds, thereby addressing the key limitations observed with conventional optimizers. We evaluate Quartz across three benchmark datasets (MNIST, Fashion-MNIST, and Arabic Character Classification) using multiple convolutional neural network (CNN) architectures, including GoogLeNet, VGGNet, and ResNet-18. The experimental results demonstrate that Quartz achieves up to 99.54% test accuracy on MNIST, 91.6% test accuracy on Fashion-MNIST, and 98.27% test accuracy on Arabic Character Classification. It consistently outperformed or matched the results achieved using state-of-the-art adaptive optimizers under identical training conditions. In terms of efficiency, Quartz reduces training time in several settings while maintaining higher accuracy. Across all the experiments, Quartz also shows statistically significant improvements (p< 0.001 in most comparisons) and smoother convergence behavior, indicating improved optimization stability. These findings highlight Quartz’s potential as a reliable and efficient tool for use in optimization strategies for a broad range of deep learning tasks.
Ahmad Raza Khan, Sarab Almuhaideb· Electronics· 0 citations
Deep neural networks often encounter non-convex optimization challenges during training due to the presence of local minima, saddle points, and complex loss surfaces. Existing optimization algorithms such as Adam and Stochastic Gradient Descent (SGD) offer complementary advantages—Adam provides faster convergence, while SGD tends to achieve better generalization. However, neither optimizer alone effectively balances both properties in non-convex settings. To address this limitation, this paper proposes a phase-switch hybrid optimization strategy that combines the strengths of Adam and SGD. The proposed method employs Adam during the initial phase of training to enable rapid convergence and efficient exploration of the loss landscape, and then switches to momentum-based SGD in the later phase to improve generalization and ensure stable convergence. The effectiveness of the proposed approach is evaluated on one benchmark dataset, MNIST dataset, under different learning rate settings. Experimental results demonstrate that the proposed method achieves performance that is superior or comparable to existing optimizers in terms of accuracy and loss minimization. These results indicate that the proposed hybrid optimization strategy provides a simple and effective solution for handling non-convex optimization problems in deep learning.
Adaptive Log-Space (AL) quantization for non-negative states is introduced and its results make state topology and update semantics first-class design constraints for optimizer quantization.
Gradient descent-based optimization algorithms are widely used in deep learning, where adaptive optimizers dynamically adjust parameter updates to improve training efficiency. However, most existing adaptive optimizers mainly rely on historical gradients while overlooking short-term gradient fluctuations. Although methods such as diffGrad and AngularGrad attempt to address this issue through gradient-difference or angle-based correction mechanisms, their performance improvements are often task-dependent, and some approaches introduce additional computational overhead due to complex vector operations. To address these limitations, this paper proposes a lightweight short-term gradient-aware mechanism based on gradient sign consistency, which improves optimization stability while maintaining low computational complexity. Based on this idea, two Adam-based optimizers, sigSignGrad and tanhSignGrad, are developed by incorporating adaptive friction coefficients constructed through Sigmoid and Tanh mappings, respectively. In addition, the proposed friction coefficient can be flexibly integrated into the Adam optimizer family as a lightweight plug-in module. Extensive experiments are conducted across multiple computer vision and natural language processing tasks, including image classification, object detection, semantic segmentation, machine translation, language modeling, and language understanding benchmarks. To evaluate the robustness and generalization capability of the proposed methods, multiple datasets and model architectures are considered within the same task settings, covering both training-from-scratch and pre-trained fine-tuning scenarios. Experimental results show that sigSignGrad achieves the best overall performance on multiple visual tasks, including all evaluated Vision Transformer benchmarks, Pascal VOC object detection, and semantic segmentation datasets. In natural language processing tasks, tanhSignGrad achieves the best BLEU scores on the IWSLT14 De-En translation task and consistently outperforms existing optimizers in language modeling experiments under both pre-trained and training-from-scratch settings. Furthermore, the proposed methods maintain computational efficiency comparable to diffGrad while providing more stable optimization behavior across diverse architectures and datasets.
Rongjian Chen, Ruyan Xiong, Xiaodong Yang· International Journal of Com...· 0 citations
On committed future-minibatch sequences, optimizer memory and near-future data order are actionable components of the training state, providing a mechanism-based criterion for when finite-horizon rather than one-step intervention is required.
Jinhui Guo· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
