In this talk I will discuss how different classes of generative models can be adapted to handle discrete random variables, and how this can be used to connect generative models to downstream tasks such as lossless compression. I will start by discussing normalizing flow models, and the challenges that arise when converting these models that are typically designed for real-valued random variables to discrete random variables. Next, I will demonstrate how denoising diffusion models with discrete state spaces have a rich design space in terms of the noising process, and how this influences the performance of the learned denoising model. Finally, I will show how denoising diffusion models can be connected to autoregressive models, and introduce an autoregressive model with a random generation order.