University of Cambridge > Talks.cam > Machine Learning Reading Group @ CUED > The unreasonable effectiveness of mathematics in large scale deep learning

  Browse A-Z

Log in

University Account External (via Google)

🔐 Log In

Information on

Subscribing to talksdetails Finding a talkdetails Adding a talkdetails Disseminating talksdetails Help and Documentationdetails

ℹ️ Information

The unreasonable effectiveness of mathematics in large scale deep learning

Download to your calendar using vCal

If you have a question about this talk, please contact James Allingham .

Recently, the theory of infinite-width neural networks led to the first technology, muTransfer, for tuning enormous neural networks that are too expensive to train more than once. For example, this allowed us to tune the 6.7 billion parameter version of GPT -3 using only 7% of its pretraining compute budget, and with some asterisks, we get a performance comparable to the original GPT -3 model with twice the parameter count. In this talk, I will explain the core insight behind this theory. In fact, this is an instance of what I call the Optimal Scaling Thesis, which connects infinite-size limits for general notions of “size” to the optimal design of large models in practice, illustrating a way for theory to reliably guide the future of AI. I’ll end with several concrete key mathematical research questions whose resolutions will have incredible impact on how practitioners scale up their NNs.

There’s no required reading for the talk but folks can look at my homepage for an overview of Tensor Programs.

This talk is part of the Machine Learning Reading Group @ CUED series.

This talk is included in these lists:

Note that ex-directory lists are not shown.

Other lists

https://talks.cam.ac.uk/show/index/187177#:~:text=Help%20and%20Documentation-,West%202%2C%20West%20Hub,-Add%20to%20your Centre for Animal Welfare & Anthrozoology Seminar Room, Department of Veterinary Medicine Seminar Room 3, Clinical School

Other talks

TBC Grand Rounds - Equine Title to be confirmed Machine Politics? Computers and AI in the History of Political Thought Title and abstract tba Cambridge RNA Club - ONLINE
 

© 2006-2025 Talks.cam, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity