PAC-Bayes Bounds on Quotient Parameter Spaces: Geometry-induced Implicit-Bias Priors
It is shown that PAC--Bayesian analysis should be performed on the quotient predictor space: pushing a prior and posterior to the quotient preserves the empirical and population Gibbs risks while removing the nonnegative KL contribution caused solely by how the two distributions differ among parameterizations of the same predictor.