Abstract

In recent years, machine learning (ML) ignited a revolution in data compression as researchers and engineers can now design codecs that learn how to encode information optimally from large datasets. These ML-based methods use deep generative models (DGM), such as variational autoencoders or diffusion models, to build a distribution over the data. DGMs generate data by simulating a sample from a simple latent distribution, such as a Gaussian, which they transform into a sample from the data distribution using a deep neural network. Hence, we can encode data by encoding the latent sample that generated it and using the DGM to reconstruct it. However, a surprising fact is that traditional methods for encoding the latent sample are suboptimal, and a far more efficient approach exists called relative entropy coding (REC).

In this talk, I will first give an overview of learned data compression and some issues it faces and use it to motivate REC . Then, I will present a simple REC algorithm, revealing a surprising equivalence between sampling and search. Finally, I will discuss the main limitations of current REC algorithms, which prevent their practical application so far and lay out some potential ways to resolve these limitations.

You can also join us on Zoom