BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:Unlikelihood-training and Back-training for robust natural languag
 e understanding - Siva Reddy\, McGill University
DTSTART:20211014T130000Z
DTEND:20211014T140000Z
UID:TALK163429@talks.cam.ac.uk
CONTACT:Marinela Parovic
DESCRIPTION:Language models are known to be good at generalization and mem
 orization. These abilities mean that a language model can be directly be u
 sed as a knowledge base\, e.g.\, a language model could easily fill the bl
 ank in the sentences "The capital of Canada is BLANK" and "BLANK is the ca
 pital of Canada\;" with Ottawa\, even if these exact syntactic constructio
 ns are never seen during training\, a task that requires both generalizati
 on and memorization. But we also observe that complex phenomena such as ne
 gation are commonly ignored by language models\, e.g.\, the model would st
 ill predict Ottawa as the answer to "The capital of Canada is not BLANK". 
 I will introduce a new training procedure and objective called "unlikeliho
 od training with reference" in order to build language models that underst
 and negation. \n\nIn the second part of the talk\, I will show that pretra
 in and fine-tune paradigm breaks in an out-of-distribution setting. For ex
 ample\, question answering and generation models trained on Natural Questi
 ons do not generalize to other domains such as education or bio-medical. I
  will introduce a new technique called back-training that exploits unsuper
 vised data in the target domains much more efficiently than self-training.
LOCATION:https://cam-ac-uk.zoom.us/j/97599459216?pwd=QTRsOWZCOXRTREVnbTJBd
 XVpOXFvdz09
END:VEVENT
END:VCALENDAR