Abstract

A system is monitored over time. At time $t$ an input $A_t$ is determined and an output $S_{t+}$ observed. The input is directly controllable by an experimenter but the output $S_{t+}$ is not.

The statement above applies to a raft of problems across multiple areas, including control, machine learning, stochastic scheduling and many others. A special case of growing interest in the biostatistical literature is the need for statistical methodology aimed at determining optimal dynamic treatment regimes from observational data. Motivation is primarily related to individualised medicine and causal inference.

Building on the advantage learning approach, we propose a modelling and estimation strategy that incorporates the regret functions of Murphy (2003) into a regression model for observed responses. Estimation is quick and diagnostics are available, meaning a variety of candidate models can be compared. We consider three issues relating to the missing data problem that is ubiquitous in practical applications: estimation, sub-optimal decisions and non-recovery. The methods are illustrated using data on patients on long-term anticoagulation treatment.