BEGIN:VCALENDAR VERSION:2.0 PRODID:-//Talks.cam//talks.cam.ac.uk// X-WR-CALNAME:Talks.cam BEGIN:VEVENT SUMMARY:Asymptotic Analysis of Deep Residual Networks - Rama Cont (Univers ity of Oxford) DTSTART:20211027T153000Z DTEND:20211027T173000Z UID:TALK164644@talks.cam.ac.uk DESCRIPTION:

Residual networks (ResNets) have display ed impressive results in pattern recognition and\, recently\, have garnere d considerable theoretical interest due to a perceived link with neural or dinary differential equations (neural ODEs). This link relies on the conve rgence of network weights to a smooth function as the number of layers inc reases. We investigate the properties of weights trained by stochastic gra dient descent and their scaling with network depth through detailed numeri cal experiments. We observe the existence of scaling regimes markedly diff erent from those assumed in neural ODE literature: one may obtain an alter native ODE limit\, a stochastic differential equation or neither of these. The scaling regime one ends up with depends on certain features of the ne twork architecture\, such as the smoothness of the activation function. Th ese findings cast doubts on the validity of the neural ODE model as an ade quate asymptotic description of deep ResNets and point to an alternative c lass of differential equations as a better description of the deep network limit. \; In the case where the scaling limit is a stochastic differe ntial equation\, the deep network limit is shown to be described by a syst em of forward-backward stochastic differential equations. Joint work with: Alain-Sam Cohen (InstaDeep Ltd)\, Alain Rossier (Oxford)\, RenYuan Xu (Un iversity of Southern California).

LOCATION:Discussion Room\, Newton Institute END:VEVENT END:VCALENDAR