BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:The K-FAC method for neural network optimization - James Martens\,
  Google Deep Mind
DTSTART:20190314T140000Z
DTEND:20190314T150000Z
UID:TALK121438@talks.cam.ac.uk
CONTACT:Alberto Bernacchia
DESCRIPTION:Second order optimization methods have the potential to be muc
 h faster than first order methods in the deterministic case\, or pre-asymp
 totically in the stochastic case. However traditional second order methods
  have proven ineffective or impractical for neural network training\, due 
 in part to the extremely high dimension of the parameter space. Kronecker-
 factored Approximate Curvature (K-FAC) is second-order optimization method
  based on a tractable approximation to the Gauss-Newton/Fisher matrix that
  exploits the special structure of neural network training objectives.  Th
 is approximation is neither low-rank nor diagonal\, but instead involves K
 ronecker-products\, which allows for efficient estimation\, storage and in
 version of the curvature matrix. In this talk I will introduce the basic K
 -FAC method for standard MLPs and then present some more recent work in th
 is direction\, including extensions to CNNs and RNNs\, both of which requi
 res new approximations to the Fisher.  For these I will provide theoretica
 lly motivated arguments\, as well as empirical results which speak to thei
 r efficacy in neural network optimization.
LOCATION:Cambridge University Engineering Department\, CBL\, BE4-38 (http:
 //learning.eng.cam.ac.uk/Public/Directions)
END:VEVENT
END:VCALENDAR