University of Cambridge > Talks.cam > CCIMI Seminars > Coordinate Descent on the Orthogonal Group for Recurrent Neural Network Training

  Browse A-Z

Log in

Googledetails Microsoftdetails

🔐 Log In

Information on

Subscribing to talksdetails Finding a talkdetails Adding a talkdetails Disseminating talksdetails Help and Documentationdetails

ℹī¸ Information

Coordinate Descent on the Orthogonal Group for Recurrent Neural Network Training

Download to your calendar using vCal

If you have a question about this talk, please contact Hamza Fawzi .

To address the poor scalability of training algorithms for orthogonal recurrent neural networks, we propose to use a coordinate descent method on the orthogonal group. This algorithm has a cost per iteration that evolves linearly with the number of recurrent states, in contrast with the cubic dependency of typical algorithms such as stochastic Riemannian gradient descent. We numerically show that the Riemannian gradient in recurrent neural network training has an approximately sparse structure. Leveraging this observation, we propose a variant of the proposed algorithm that relies on Gauss-Southwell coordinate selection. Experiments on a benchmark recurrent neural network training problem show that the proposed approach is a very promising step towards the training of orthogonal recurrent neural networks with big architectures.

Join Zoom Meeting https://maths-cam-ac-uk.zoom.us/j/93776043287?pwd=UDIrNDdkeUU1NmFtZXpNUzd6ZjRrdz09 Meeting ID: 937 7604 3287 Passcode: p1Co4skf

This talk is part of the CCIMI Seminars series.

This talk is included in these lists:

Note that ex-directory lists are not shown.

Other lists

Webinar & LT2, Computer Laboratory, William Gates Building. Engelsberg Room, Centre for Geopolitics, Fitzwilliam House, 32 Trumpington Street, Cambridge Average on weekdays: 8962.3175853 kWh

Other talks

Grand Rounds - Equine Unveiling the curtain with ukubuyisa or go lata: complicated roles of museums and nation states in the cultural repatriation discourse and practice in southern Africa title abstract tba Title TBC Morning Break The uplift and erosion history of the Gamburtsev Subglacial Mountains and their role in the formation of the East Antarctic Ice Sheet
 

Š 2006-2025 Talks.cam, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity