S. Akhatkulov

Control Parameter Dynamics in Neural Network Optimization: An Analysis of Learning Rate Strategies for Sequential Data Modeling

The learning rate (LR) is the single most influential hyperparameter governing the convergence and stability of stochastic gradient descent (SGD). While numerous heuristic schedules for LR decay and warmup have been proposed to improve generalization, their underlying mathematical behavior as discrete control parameter...