Painting with Probability: A Narrative Review of Deep Generative Models from Variational Autoencoders to Latent Diffusion

Abstract

Deep generative models---neural networks that learn data's distribution and sample from it---matured from variational autoencoders' latent geometry to diffusion models' state-of-the-art images. This article presents a narrative review of that arc's canonical line: Kingma and Welling's 2013 auto-encoding variational Bayes, Goodfellow and colleagues' 2014 generative adversarial networks, Mirza and Osindero's 2014 conditional GANs, Rezende and Mohamed's 2015 normalizing flows, Sohl-Dickstein and colleagues' 2015 nonequilibrium thermodynamics, Arjovsky and colleagues' 2017 Wasserstein GANs, Karras and colleagues' 2019 style-based generator, Ho and colleagues' 2020 denoising diffusion, Ramesh and colleagues' 2021 text-to-image generation, Dhariwal and Nichol's 2021 diffusion-beats-GANs result, Nichol and Dhariwal's 2021 improved diffusion, and Rombach and colleagues' 2022 latent diffusion. The synthesis is organized around three themes: latent foundations, in which autoencoders and flows made sampling principled; adversarial training, in which games between generator and discriminator produced realism; and diffusion's rise, in which denoising trajectories conquered synthesis. It is concluded that generative modeling's decade ran from likelihood's compromise to sampling's triumph---and that latent diffusion is the field's new foundation.