BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY: Implicit Regularization in Deep Learning -  Jezabel Garcia\, Albe
 rto Bernacchia (MediaTek Research)
DTSTART:20211027T100000Z
DTEND:20211027T113000Z
UID:TALK165115@talks.cam.ac.uk
CONTACT:Elre Oldewage
DESCRIPTION:Empirically\, it has been observed that overparameterized neur
 al networks trained by stochastic gradient descent (SGD) generalize well\,
  even in absence of any explicit regularization. Because of overparameteri
 zation\, there exist minima of the training loss which generalize poorly\,
  but such bad minima are never encountered in practice. In recent years\, 
 a growing body of work suggests that the optimizer (SGD or similar) implic
 itly regularizes the training process and leads towards good minima that g
 eneralize well. In this presentation\, we review three (non-exclusive) the
 ories that aim at quantifying this effect: 1) Minibach noise in SGD avoids
  sharp minima that generalize poorly\, 2) Gradient descent finds solutions
  with minimum norm\, 3) SGD is equivalent to regularized gradient flow. Th
 ese theories may improve our understanding of optimization and generalizat
 ion in overparameterized models.\n\nReadings:\n\nhttps://arxiv.org/abs/161
 1.03530\n\nhttps://arxiv.org/abs/1710.06451\n\nhttps://arxiv.org/abs/2002.
 09277\n\nhttps://arxiv.org/abs/1905.13655\n\nhttps://arxiv.org/abs/2101.12
 176\n\nZoom link: https://eng-cam.zoom.us/j/82019956685?pwd=WUNSVVcrdC9IZG
 xQOHFhSThjUjd2dz09
LOCATION: Cambridge University Engineering Department \,LR3A
END:VEVENT
END:VCALENDAR