How Inefficient Is Natural Gradient Descent? From Exact Optimality to \Theta ( \sqrt{ \log d } ) Divergence
Natural gradient descent requires asymptotically at least \(R\) times as many steps as an optimizer following the Fisher--Rao geodesic, a practical computational cost under a per-step Fisher-chord budget.