Abstract

Please join us for our fortnightly journal club online via zoom where two presenters will jointly present a topic together. The next topic is ‘Mental simulation in human reinforcement learning’ presented by Kris Jensen and Georgia Turner.

Zoom information: https://us02web.zoom.us/j/84958321096?pwd=dFpsYnpJYWVNeHlJbEFKbW1OTzFiQT09 Meeting ID: 841 9788 6178 Passcode: 659046

Summary:

Both animals and artificial agents are capable of learning from their interactions with an environment in a reinforcement learning setting. However, humans and other animals are also known to learn ‘offline’ from prior experiences which can be flexibly combined with information from more recent events. In this journal club, we present the ‘Dyna’ algorithm (Sutton 1990) which formalizes such offline reinforcement learning via ‘mental simulation’ of prior events. We also describe a normative framework for identifying what prior experience should be replayed at any given moment in time to maximize performance (Mattar & Daw 2018), and we present experimental evidence for such experience replay and its structure (Gershman et al. 2014; Liu & Mattar et al. 2021).

Sutton, R. S. (1991). Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2(4), 160-163.

Gershman, S. J., Markman, A. B., & Otto, A. R. (2014). Retrospective revaluation in sequential decision making: A tale of two systems. Journal of Experimental Psychology: General, 143(1), 182.

Mattar, M. G., & Daw, N. D. (2018). Prioritized memory access explains planning and hippocampal replay. Nature neuroscience, 21(11), 1609-1617.

Liu, Y., Mattar, M. G., Behrens, T. E., Daw, N. D., & Dolan, R. J. (2021). Experience replay is associated with efficient nonlocal learning. Science, 372(6544).