Abstract

In many machine learning applications, complicated parametric density models such as Deep Neural Networks are favoured due to their flexibility and expressiveness. However, unlike classic density models, these density functions cannot be easily normalised. Thus, Maximum Likelihood Estimation (MLE) cannot be easily applied for model parameter estimation. If the dimensionality of the input variable is high, MCMC based MLE may also fail. In the machine learning community, some Stein Identity based methods have risen in popularity due to their ability to measure the “goodness of fit” of unnormalisable models. In the first part of the talk [1], we study the performance of a parameter estimator using Stein’s identity. Particularly, we construct a Stein density ratio estimator, which estimates the ratio function between a data distribution and a model distribution. Then we minimise the fitted likelihood ratio to estimate model parameters. In the second part of the talk [2], I discuss a specific type of unnormalised density model: Truncated densities. We show how an augmented score matching estimator can be applied to estimate parameters of density models with a complex truncation domain (such as a polytope in R^2).

[1] Liu, S., Kanamori, T., Jitkrittum, W., Chen, Y; Fisher Efficient Inference of Intractable Models; arxiv:1805.07454, 2019. [2] Liu, S., Kanamori, T., Estimating Density Models with Complex Truncation Boundaries; arXiv:1910.03834, 2019.