Abstract

Scientists run experiments to distinguish between competing hypotheses, but how do we select the best experiment to run? The answer is often non-trivial, as there are usually many possible experiments but limited time and resources. I describe a system for Bayesian optimal experiment design (OED) based on probabilistic programming languages (PPLs): given hypotheses encoded as PPL models and an explicit definition of the experiment space, OED automates the search for experiments with high expected information gain. Additionally, I describe “adaptive OED ”, a framework for active learning: by updating our prior beliefs on hypotheses with the observed response to an experiment, OED can suggest further experiments to continue to tease apart hypotheses. I apply this system to two domains in cognitive psychology—sequence prediction and causal knowledge—and demonstrate that adaptive OED performs better than standard OED and other naive experiment selection procedures.