Abstract

Please join us for our fortnightly journal club online via zoom where two presenters will jointly present a topic together. The next topic is ‘Analysis of RNNs’ presented by Jake Stroud and David Liu.

Zoom information: https://us02web.zoom.us/j/84958321096?pwd=dFpsYnpJYWVNeHlJbEFKbW1OTzFiQT09 Meeting ID: 849 5832 1096 Passcode: 506576

Summary: Recurrent neural networks (RNNs) have been successfully used to aid our understanding many different brain regions from visual cortex, to the basal ganglia, hippocampus, and frontal cortices. Historically, these network models (and in particular the connections between neurons in the network) were hand crafted so that the model displayed the desired behaviour (e.g. Hopfield networks). Recently, with the advent of new machine learning techniques, it has become increasingly more common for researchers to instead train RNNs on a given task and then subsequently reverse-engineer the network to understand how it solved the task. In this journal club, we will present the techniques that allow us to uncover the mechanisms that trained RNNs use to solve tasks. In particular, we will show why fixed points analyses, and local linearizations around fixed points, are central to understanding the dynamics of RNNs. We will then cover a recent pre-print that uses these approaches to understand how a single RNN performs multiple different tasks.

Key papers:

Opening the black box: Low-dimensional dynamics in high-dimensional recurrent neural networks, David Sussillo and Omri Barak, Neural Computation, 2013.

Neural circuits as computational dynamical systems, David Sussillo, Current Opinion in Neurobiology, 2014.

Flexible multitask computation in recurrent networks utilizes shared dynamical motifs, Laura Driscoll, Krishna Shenoy, David Sussillo, BioRxiv, 2022