BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:Depth Without the Magic: Inductive Bias of Natural Gradient Descen
 t - Anna Kerekes (Centre for Mathematical Sciences)
DTSTART:20211207T090000Z
DTEND:20211207T100000Z
UID:TALK166753@talks.cam.ac.uk
DESCRIPTION:In gradient descent\, changing how we parametrize the model ca
 n lead to drastically different optimization trajectories\, giving rise to
  a surprising range of meaningful inductive biases: identifying sparse cla
 ssifiers or reconstructing low-rank matrices without explicit regularizati
 on. This implicit regularization has been hypothesised to be a contributin
 g factor to good generalization in deep learning. However\, natural gradie
 nt descent is approximately invariant to reparameterization\, it always fo
 llows the same trajectory and finds the same optimum. The question natural
 ly arises: What happens if we eliminate the role of parameterization\, whi
 ch solution will be found\, what new properties occur? We characterize the
  behaviour of natural gradient flow in deep linear networks for separable 
 classification under logistic loss and deep matrix factorization. Some of 
 our findings extend to nonlinear neural networks with sufficient but finit
 e over-parametrization. We demonstrate that there exist learning problems 
 where natural gradient descent fails to generalize\, while gradient descen
 t with the right architecture performs well.
LOCATION:Seminar Room 2\, Newton Institute
END:VEVENT
END:VCALENDAR