Generative Modeling of Discrete Data Using Geometric Latent Subspaces

Abstract

We propose a geometric latent-subspace framework for generative modeling of discrete data. Specifically, we introduce latent subspaces in the exponential parameter space of product manifolds of categorical distributions as a novel approach to learning low-dimensional representations of high-dimensional discrete data. The resulting low-dimensional latent space captures statistical dependencies and removes redundant degrees of freedom among the categorical variables. We equip the parameter domain with a Riemannian geometry such that the latent subspace and induced data manifold are related isometrically, enabling consistent flow matching. Exploiting this structure, we propose a geometry-aware dimensionality reduction objective, called geometric PCA (GPCA), which we formulate as a regularized cross-entropy minimization that encourages small Riemannian distances between the data and their reconstructions. In particular, under the induced geometry, geodesics correspond to straight lines in the latent parameter space, allowing flow matching to be performed directly in reduced coordinates. Empirical results show that low-dimensional latent representations suffice to accurately model high-dimensional discrete data and enable substantially more computationally efficient flow matching.