BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:A mean-field theory of lazy training in two-layer neural nets: ent
 ropic regularization and controlled McKean-Vlasov dynamics - Maxim Raginsk
 y -  University of Illinois at Urbana-Champaign
DTSTART:20210713T140000Z
DTEND:20210713T150000Z
UID:TALK161365@talks.cam.ac.uk
CONTACT:Francisco Vargas
DESCRIPTION:*Paper:*\n\nTalk is based on "this":https://arxiv.org/abs/2002
 .01987 paper.\n\n*Abstract:*\n\nWe consider the problem of universal appro
 ximation of functions by two-layer neural nets with random weights that ar
 e "nearly Gaussian" in the sense of Kullback-Leibler divergence. This prob
 lem is motivated by recent works on lazy training\, where the weight updat
 es generated by stochastic gradient descent do not move appreciably from t
 he i.i.d. Gaussian initialization. We first consider the mean-field limit\
 , where the finite population of neurons in the hidden layer is replaced b
 y a continual ensemble\, and show that our problem can be phrased as globa
 l minimization of a free-energy functional on the space of probability mea
 sures over the weights. This functional trades off the L2 approximation ri
 sk against the KL divergence with respect to a centered Gaussian prior. We
  characterize the unique global minimizer and then construct a controlled 
 nonlinear dynamics in the space of probability measures over weights that 
 solves a McKean--Vlasov optimal control problem. This control problem is c
 losely related to the Schrödinger bridge (or entropic optimal transport) 
 problem\, and its value is proportional to the minimum of the free energy.
  Finally\, we show that SGD in the lazy training regime (which can be ensu
 red by jointly tuning the variance of the Gaussian prior and the entropic 
 regularization parameter) serves as a greedy approximation to the optimal 
 McKean--Vlasov distributional dynamics and provide quantitative guarantees
  on the L2 approximation error.\n\n*Website:* https://maxim.ece.illinois.e
 du/\n\nPart of ML@CL Seminar Series in topics relevant to machine learning
  and statistics.\n\nJoin Zoom Meeting\nhttps://us02web.zoom.us/j/478429647
 1?pwd=SE8vc1BvWldlZnc1YUNwK3Q1dHpodz09\nMeeting ID: 478 429 6471\nPasscode
 : 345261
LOCATION:https://us02web.zoom.us/j/4784296471?pwd=SE8vc1BvWldlZnc1YUNwK3Q1
 dHpodz09
END:VEVENT
END:VCALENDAR