Foundations of the Scaling Law
TL;DR
This technical note explains three mechanisms behind the exponents of dense-to-equivariant excess-loss ratios, which cover visual resolution, conformational coverage and correlated-sample precision.
Abstract
Empirical scaling laws relate loss to parameters $N$, training tokens $D$ and compute $C$. This technical note explains three mechanisms behind their exponents. Local approximation bounds function error at rate $N_e^{-\gamma s/d}$, with effective parameters $N_e$, smoothness $s$, reuse exponent $\gamma$ and dimension $d$; the loss function determines the risk bound. Independent feature coverage gives $\alpha_D=(\beta-1)/\zeta$ from predictive-weight decay $\beta$ and occurrence rarity $\zeta$. Separable-loss optimization determines training allocations and deployment crossovers. Nonlinear scoring can produce apparent emergent abilities, while grokking concerns delayed generalization and verification constrains test-time scaling. For regular molecular configurations, the $\mathrm{SE}(3)$ quotient gives $d_{\mathrm{int}}=d-6$ and relative approximation-exponent gain $6/d_{\mathrm{int}}$. Optimized dense-to-equivariant excess-loss ratios scale as $C^{p_{\mathrm{eq}}-p_{\mathrm{dense}}}$, where $p_j$ are compute-loss exponents and the prefactor depends on loss amplitudes and FLOP costs. Domain examples also cover visual resolution, conformational coverage and correlated-sample precision.
