University of Cambridge > Talks.cam > Language Technology Lab Seminars > Variational Smoothing in Recurrent Neural Network Language Models

  Browse A-Z

Log in

Googledetails Microsoftdetails

🔐 Log In

Information on

Subscribing to talksdetails Finding a talkdetails Adding a talkdetails Disseminating talksdetails Help and Documentationdetails

ℹī¸ Information

Variational Smoothing in Recurrent Neural Network Language Models

Download to your calendar using vCal

If you have a question about this talk, please contact Qianchu Liu .

In this talk, we present a new theoretical perspective of data noising in recurrent neural network language models (Xie et al., 2017). We show that each variant of data noising is an instance of Bayesian recurrent neural networks with a particular variational distribution (i.e., a mixture of Gaussians whose weights depend on statistics derived from the corpus such as the unigram distribution). We use this insight to propose a more principled method to apply at prediction time and propose natural extensions to data noising under the variational framework. In particular, we propose variational smoothing with tied input and output embedding matrices and an element-wise variational smoothing method. We empirically verify our analysis on two bench-mark language modeling datasets and demonstrate performance improvements over existing data noising methods.

This talk is part of the Language Technology Lab Seminars series.

This talk is included in these lists:

Note that ex-directory lists are not shown.

Other lists

Zaflaver Bateman Room, Gonville & Caius College Lecture Theatres - LT1, Cambridge University Department of Engineering, Inglis Building

Other talks

The LMC-SMC System: A New Laboratory for Galaxy Evolution, Dynamics and Dark Matter Max Perutz Lecture - Title TBC Zooming in on the assembly of galaxies in the early Universe Morning Tea and Coffee Title to be confirmed Sheref Mansy, topic TBA
 

Š 2006-2025 Talks.cam, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity