University of Cambridge > Talks.cam > Machine Learning Reading Group @ CUED > A Tutorial on Algorithmic Information Theory in Modern ML

  Browse A-Z

Log in

University Account External (via Google)

🔐 Log In

Information on

Subscribing to talksdetails Finding a talkdetails Adding a talkdetails Disseminating talksdetails Help and Documentationdetails

ℹ️ Information

A Tutorial on Algorithmic Information Theory in Modern ML

Download to your calendar using vCal

If you have a question about this talk, please contact Xianda Sun .

This tutorial explores how ideas from algorithmic information theory connect to modern machine learning through three recent papers. We begin with Solomonoff induction—the theoretically optimal but uncomputable predictor—and show how neural networks can approximate it by training on Universal Turing Machine data (Grau-Moya et al., 2024). We then establish the formal foundations by examining Kolmogorov complexity and its connections to compression and randomness in images, exploring how the Solomonoff prior helps us understand what makes images “realistic” and guides the design of better generative models and anomaly detectors (Theis, 2024). Finally, we demonstrate these principles at scale, deriving non-vacuous generalization bounds for large language models with billions of parameters through compression-based analysis using the SubLoRA technique (Lotfi et al., 2024). No prior background in algorithmic information theory required—we’ll build intuition from first principles while connecting to familiar ML concepts throughout.

Papers:
  1. Learning Universal Predictors (Grau-Moya et al., 2024) – https://arxiv.org/abs/2401.14953
  2. What Makes an Image Realistic? (Theis, 2024) – https://arxiv.org/abs/2403.04493
  3. Non-Vacuous Generalization Bounds for Large Language Models (Lotfi et al., 2024) – https://arxiv.org/abs/2312.17173

This talk is part of the Machine Learning Reading Group @ CUED series.

This talk is included in these lists:

Note that ex-directory lists are not shown.

Other lists

Teams Livestream & Cavendish Laboratory, Pippard Lecture Theatre MCR Murray Edwards Cambridge University Engineering Department, LT2

Other talks

TBC Welcome Reception and Poster Session Polar Oceans Seminar Talk Title to be confirmed Animating urbanism: the demotic multiplicities of seeing cities digitally Title to be confirmed
 

© 2006-2025 Talks.cam, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity