BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY: Randomized Automatic Differentiation - Deniz Oktay Princeton
DTSTART:20210302T150000Z
DTEND:20210302T160000Z
UID:TALK157753@talks.cam.ac.uk
CONTACT:96082
DESCRIPTION:The successes of deep learning\, variational inference\, and m
 any other fields have been aided by specialized implementations of reverse
 -mode automatic differentiation (AD) to compute gradients of mega-dimensio
 nal objectives. The AD techniques underlying these tools were designed to 
 compute exact gradients to numerical precision\, but modern machine learni
 ng models are almost always trained with stochastic gradient descent. Why 
 spend computation and memory on exact (minibatch) gradients only to use th
 em for stochastic optimization? In this talk\, I give a quick overview of 
 basic concepts in modern AD and talk about our work on Randomized Automati
 c Differentiation (RAD)\, which is a framework that allows unbiased gradie
 nt estimates to be computed with reduced memory in return for variance. In
  the work\, we introduce a general approach for RAD\, examine limitations 
 of the general case\, and develop specialized RAD strategies exploiting pr
 oblem structure in case studies. We develop RAD techniques for a variety o
 f simple neural network architectures\, and show that for a fixed memory b
 udget\, RAD converges in fewer iterations than using a small batch size fo
 r feedforward networks\, and in a similar number for recurrent networks. W
 e also show that RAD can be applied to scientific computing\, and use it t
 o develop a low-memory stochastic gradient method for optimizing the contr
 ol parameters of a linear reaction-diffusion PDE representing a fission re
 actor.
LOCATION:https://us02web.zoom.us/j/86285792868?pwd=UGJFeit5RVozOTdqUTdGeEF
 XNlk1Zz09
END:VEVENT
END:VCALENDAR