Abstract

For many modern statistical machine learning problems, model misspecification is pervasive and impactful. In particular, inferences are not robust, and uncertainty quantification becomes brittle. These issues are exacerbated by nonparametric models like Gaussian Processes. While distance-based estimation is a powerful remedy for this setting, previously proposed distances between conditional distributions are intractable. To resolve this, we introduce a computationally tractable distance on the space of conditional probability distributions we call expected maximum conditional mean discrepancy. The theoretical properties of the resulting distance-based estimator are investigated in detail. While the estimator is of general interest, we focus on its application as a robust empirical Bayes estimator in Gaussian Process models. Specifically, we demonstrate that it produces reliable uncertainty quantification for regression problems, computer model emulation, and Bayesian optimisation.