Dynamic Regime Maps of Neural Network Training Under the Adam Optimizer: An Observable-Based Empirical Analysis

Abstract

Adaptive optimization algorithms are fundamental to modern deep learning; however, the global organization of neural-network training regimes induced by optimizer hyperparameters remains insufficiently understood. In particular, the influence of the Adam moment coefficients on the stability and qualitative behavior of the learning process has not been systematically investigated through parameter-space regime mapping. In this work, we introduce an observable-based empirical regime-mapping framework for analysing neural-network training under the Adam optimizer in the two-dimensional hyperparameter space defined by the exponential decay coefficients of the first and second gradient moments, (β 1 , β 2 ). Neural-network optimization is treated as an iterative parameter-update process evolving in a high-dimensional parameter space, while its behavior is characterized through low-dimensional observable fields derived from neuron-wise training-error dynamics. Rather than relying on a single observable, the proposed framework combines seven complementary empirical descriptors that characterize regime organization, temporal stability, alignment, anisotropy, and training evolution. Experiments are performed on datasets of increasing complexity, including printed-digit patterns, Fashion-MNIST, CIFAR-10, and CIFAR-100, using multilayer neural-network architectures of varying width and depth. The resulting regime maps reveal fragmented hyperparameter landscapes containing stable, oscillatory, slow-learning, and irregular observable regimes. Increasing dataset complexity and network capacity is generally associated with smaller coherent stable regions and increased sensitivity to the Adam moment coefficients. The geometric complexity of the regime boundaries is quantified using box-counting analysis. The estimated dimensions approach D ≈ 1.9 for several investigated configurations, indicating highly irregular and nearly space-filling boundaries at the available numerical resolution. These values are interpreted as empirical measures of boundary complexity rather than as evidence of exact mathematical fractality. Additional robustness experiments performed using 300, 500, and 1000 optimizer steps demonstrate that the large-scale organization of all seven observable fields remains largely preserved, whereas the principal changes are concentrated near transition boundaries. This persistence indicates that the detected regime structures are reproducible and are not solely artifacts of short optimization histories. The proposed methodology provides an empirical computational framework for visualizing and comparing optimization regimes in the Adam hyperparameter space. It facilitates the identification of comparatively stable hyperparameter regions and offers a complementary observable-based perspective on the complex behavior of adaptive neural-network optimization.