Abstract

Sequential Experimental Design (SED) aims to adaptively collect data that most efficiently reduce uncertainty in a statistical model of interest in a multi-stage context, where each stage’s choice depends on the knowledge gained from previous ones. Although this is a classic task in many applied disciplines, recent advances in Reinforcement Learning (RL) offer interesting new perspectives for approaching this challenge. In this talk, I will first discuss how the Bayes-Adaptive Markov Decision Processes (BAMDPs) framework provides a principled way to balance exploration and exploitation in RL when key parameters are uncertain, and how it naturally lends itself to model multi-stage SED through a parametrized “sampling policy” that selects actions/experiments based on the current context. Building on ideas from Model-Based Reinforcement Learning (MBRL), I will then present a method for picking design choices in SED by “looking ahead” and maximizing a Cumulative Expected Information Gain (C-EIG) objective over H-steps, which can be approximated in the context of MBRL -based planning via a measure of disagreement among an ensemble of probabilistic dynamics models. By explicitly modelling how each design choice affects downstream data-collection, this approach avoids myopic behaviour and enables the selection of actions/experiments that are most informative in expectation not just immediately, but across an entire H-steps sequence of trials.