BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Ne
 tworks - Ryota Tomioka (MSR)
DTSTART:20171107T140000Z
DTEND:20171107T150000Z
UID:TALK86351@talks.cam.ac.uk
CONTACT:Damon Wischik
DESCRIPTION:New types of compute hardware in development and entering the 
 market hold the promise of revolutionizing deep learning in a manner as pr
 ofound as GPUs. However\, existing software frameworks and training algori
 thms for deep learning have yet to evolve to fully leverage the capability
  of the new wave of silicon. In particular\, models that exploit structure
 d input via complex and instance-dependent control flow are difficult to a
 ccelerate using existing algorithms and hardware that typically rely on mi
 nibatching. We present an asynchronous model-parallel (AMP) training algor
 ithm that is specifically motivated by training on networks of interconnec
 ted devices. Through an implementation on multi-core CPUs\, we show that A
 MP training converges to the same accuracy as conventional synchronous tra
 ining algorithms in a similar number of epochs\, but utilizes the availabl
 e hardware more efficiently\, even for small minibatch sizes\, resulting i
 n shorter overall training times. Our framework opens the door for scaling
  up a new class of deep learning models that cannot be efficiently trained
  today.\n
LOCATION:Centre for Mathematical Sciences\, MR2
END:VEVENT
END:VCALENDAR