Oscillation-Aware Deep Learning Optimization Using Second-Order Velocity Derivatives
Abstract
Gradient-based optimization methods are fundamental to deep learning but often suffer from oscillatory behavior, slow convergence, and instability in non-convex loss landscapes. This paper introduces an oscillation-aware optimization framework based on second-order velocity derivatives, referred to as the second-order velocity (SOV) optimizer. By interpreting neural network parameter updates as a discretetime motion process, the proposed method derives a second-order velocity difference to capture acceleration-like dynamics and detect oscillations during training. A moment-based estimation mechanism is employed to stabilize this signal, and an adaptive scaling factor is formulated to regulate the effective learning rate based on oscillation intensity. This enables reduced step sizes in unstable regions while maintaining efficient convergence in smoother regions. Experimental evaluations on MNIST, Fashion-MNIST, and CIFAR-10 demonstrate that the proposed approach achieves competitive accuracy with improved stability compared to conventional optimizers such as stochastic gradient descent space(SGD), root mean square propagation(RMSProp), and adaptive moment estimation(Adam).