Equivalence Between Nested Gibbs Measures and Log-Linear Combinations of Gibbs Measures

Abstract

In this paper, three operations on Gibbs probability measures are studied. The first operation, often referred to as renormalization, takes one Gibbs probability measure and generates a new Gibbs measure by normalizing a power of its density. This normalization has a twofold effect: it changes the regularization factor and concentrates the support within a subset of the original support. Interestingly, these effects can be independently controlled by different parameters. The second operation consists of a normalized log-linear combination of the densities of Gibbs probability measures. The third operation takes two Gibbs probability measures and changes the reference measure of the latter with the former. Hence, the former is said to be "nested" within the latter, yielding a new Gibbs probability measure. The resulting measures from the second and third operations are also Gibbs probability measures and are shown, respectively, to solve optimization problems involving the expectations of linear combinations of the objective functions of the given measures, subject to a relative entropy regularization. These optimization problems differ exclusively in the coefficients of the linear combinations. This leads to the conclusion that there exists a set of parameters for which nesting one Gibbs probability measure into another has the same effect as log-linearly combining them. These operations have relevant applications in statistical learning. As an example, a one-shot federated learning system in which clients send their locally trained Gibbs algorithms to the server for log-linear combination is shown to achieve the same performance as a Gibbs algorithm trained upon the aggregation of all local training datasets.