Abstract

Please join us for our fortnightly journal club online via zoom where two presenters will jointly present a topic together.

Join Zoom Meeting https://us02web.zoom.us/j/89883818695?pwd=TTVVYitVT1VXMHZ5UXIwTFE4ZCtMQT09

Meeting ID: 898 8381 8695

Passcode: 668906

The next topic is ‘computational models of synaptic complexity’ presented by Gido Van de Jen and Xizi Li.

Biological synapses are highly complex with a multitude of molecular signalling pathways. Yet, in classical models of synaptic plasticity as well as in deep neural networks, synaptic efficacy is typically modelled as a single scalar value. Moreover, theoretical considerations alone diminish such representations, as neural networks with scalar synapses have strikingly limited memory capacity once you assume a finite number of distinguishable levels of synaptic strengths. In this journal club, we shall first discuss theoretical studies using the “ideal observer formalism” that show that the memory capacity of complex synapses can be substantially higher than that of simple scalar synapses with the realistic assumption of limited precision in synaptic strength ([1] and related work). Then, we explore whether synaptic complexity can be beneficial in deep neural networks (DNNs). Specifically, we introduce how the complex synapse model from [1] is applied “out-of-the-box” to reinforcement learning with Deep Q-Networks [2], and how a slightly different version of synaptic complexity substantially reduces catastrophic forgetting in DNNs [3].

[1] Benna, M. K., & Fusi, S. (2016). Computational principles of synaptic memory consolidation. Nature neuroscience 19(12): 1697-1706, https://www.nature.com/articles/nn.4401

[2] Kaplanis, C., Shanahan, M., & Clopath, C. (2018). Continual Reinforcement Learning with Complex Synapses. International Conference on Machine Learning, https://arxiv.org/abs/1802.07239

[3] Zenke, F., Poole, B., & Ganguli, S. (2017). Continual learning through synaptic intelligence. International Conference on Machine Learning, https://arxiv.org/abs/1703.04200