Hierarchy-Aligned Learning Rates for Vision Networks

Abstract

Learning rate (LR) initialization and decay remain important factors in the optimization of deep vision networks. Although these models exhibit a clear hierarchical structure, training typically starts from a single global learning rate, with little explicit consideration of stage depth. This paper investigates a simple depth-aware alternative. We propose Asynchronous Learning Rate (ALR), which assigns depth-dependent initial learning rates to network modules according to their topological depth, and Smoothed Synchronous Decay (SSD), which coordinates the subsequent decay of heterogeneous parameter groups. A linear depth rule is adopted as a low-complexity, monotonic parameterization that is straightforward to implement across hierarchical architectures. The proposed strategy is evaluated on image classification and semantic segmentation benchmarks, including CIFAR-100, Mini-ImageNet, ImageNet-1K, Pascal VOC 2012, and LiTS. Additional ResNet-50 evaluations under benchmark-specific training protocols show that depth-scaled ALR improves Top-1 accuracy from 77.618% to 78.736% on native CIFAR-100 across five seeds and from 76.33% to 78.58% on ImageNet-1K. The results indicate that ALR generally outperforms uniform learning rate initialization, whereas SSD is most effective when combined with depth-aware initialization, particularly when layer-wise learning rates would otherwise follow misaligned decay trajectories. These findings suggest that a simple hierarchy-aware learning rate design can serve as an effective optimization refinement for hierarchical vision models.