Does Step Law Transfer to Small-Scale Language Models? An Empirical Recalibration Below 59M Parameters
Direct transfer of Step Law systematically overestimates the optimal learning rate: the median ratio eta_SL / eta* is approximately 4.0x, with a range of 2.4x to 6.6x.
Egor Romanyukov, Timofey Novikov, Timur Shokarov et al.
· 0 citations