BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:Distributed stochastic optimization for deep learning - Sixin Zhan
 g (NYU)
DTSTART:20160614T090000Z
DTEND:20160614T100000Z
UID:TALK66515@talks.cam.ac.uk
CONTACT:Louise Segar
DESCRIPTION:We study the problem of how to distribute the training of larg
 e-scale deep learning models in the parallel computing environment. We pro
 pose a new distributed stochastic optimization method called Elastic Avera
 ging SGD (EASGD). We analyze the convergence rate of the EASGD method in t
 he synchronous scenario and compare its stability condition with the exist
 ing ADMM method in the round-robin scheme. An asynchronous and momentum va
 riant of the EASGD method is applied to train deep convolutional neural ne
 tworks for image classification on the CIFAR and ImageNet datasets. Our ap
 proach accelerates the training and furthermore achieves better test accur
 acy. It also requires a much smaller amount of communication than other co
 mmon baseline approaches such as the DOWNPOUR method.\n\nWe then investiga
 te the limit in speedup of the initial and the asymptotic phase of the min
 i-batch SGD\, the momentum SGD\, and the EASGD methods. We find that the s
 pread of the input data distribution has a big impact on their initial con
 vergence rate and stability region. We also find a surprising connection b
 etween the momentum SGD and the EASGD method with a negative moving averag
 e rate. A non-convex case is also studied to understand when EASGD can get
  trapped by a saddle point.\n\nFinally\, we scale up the EASGD method by u
 sing a tree structured network topology. We show empirically its advantage
  and challenge. We also establish a connection between the EASGD and the D
 OWNPOUR method with the classical Jacobi and the Gauss-Seidel method\, thu
 s unifying a class of distributed stochastic optimization methods.\n\n(See
  https://arxiv.org/abs/1605.02216)\n\n
LOCATION:Engineering Department\, CBL Room BE-438
END:VEVENT
END:VCALENDAR