BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:A fast and simple algorithm for training neural probabilistic lang
 uage models - Andriy Mnih\, University College London
DTSTART:20121026T110000Z
DTEND:20121026T120000Z
UID:TALK40515@talks.cam.ac.uk
CONTACT:Ekaterina Kochmar
DESCRIPTION:In spite of their superior performance\, neural probabilistic 
 language\nmodels (NPLMs) remain far less widely used than n-gram models du
 e to\ntheir notoriously long training times\, which are measured in weeks 
 even\nfor moderately-sized datasets. Training NPLMs is computationally exp
 ensive\nbecause they are explicitly normalized\, which leads to having to 
 consider\nall words in the vocabulary when computing the log-likelihood gr
 adients.\n\nWe propose a fast and simple algorithm for training NPLMs base
 d on\nnoise-contrastive estimation\, a newly-introduced procedure for esti
 mating\nunnormalized continuous distributions. We investigate the behaviou
 r of\nthe algorithm on the Penn Treebank corpus and show that it reduces t
 he\ntraining times by more than an order of magnitude without affecting th
 e\nquality of the resulting models. The algorithm is also more efficient a
 nd\nmuch more stable than importance sampling because it requires far fewe
 r\nnoise samples to perform well.\n\nWe demonstrate the scalability of the
  proposed approach by training\nseveral neural language models on a 47M-wo
 rd corpus with a 80K-word\nvocabulary\, obtaining state-of-the-art results
  on the Microsoft Research\nSentence Completion Challenge dataset.\n\nJoin
 t work with Yee Whye Teh
LOCATION:FW26\, Computer Laboratory
END:VEVENT
END:VCALENDAR