How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents
Scaling laws predict how loss decreases with increases in computation. We show, contrary to conventional wisdom, that architectural interventions can modify scaling exponents in pre-training, leading to power-law improvements in performance as computation increases. As an anchoring point, we consider the architectural...
Zi-Xi Chen, Akshay Vegesna, Samip Dahal et al.
· 2 citations