BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:Beat the AI: Investigating Adversarial Human Annotations for Readi
 ng Comprehension - Max Bartolo (UCL)
DTSTART:20200214T120000Z
DTEND:20200214T130000Z
UID:TALK139591@talks.cam.ac.uk
CONTACT:James Thorne
DESCRIPTION:Innovations in annotation methodology have been a propellant f
 or Reading Comprehension (RC) datasets and models. One recent trend to cha
 llenge current RC models is to involve a model in the annotation process: 
 humans create questions adversarially\, such that the model fails to answe
 r them correctly. In this work we investigate this annotation approach and
  apply it in three different settings\, collecting a total of 36\,000 samp
 les with progressively stronger models in the annotation loop. This allows
  us to explore questions such as the reproducibility of the adversarial ef
 fect\, transfer from data collected with varying model-in-the-loop strengt
 hs\, and generalisation to data collected without a model. We find that tr
 aining on adversarially collected samples leads to strong generalisation t
 o non-adversarially collected datasets\, yet with progressive deterioratio
 n as the model-in-the-loop strength increases. Furthermore we find that st
 ronger models can still learn from datasets collected with substantially w
 eaker models in the loop: When trained on data collected with a BiDAF mode
 l in the loop\, RoBERTa achieves 36.0F1 on questions that it cannot answer
  when trained on SQuAD - only marginally lower than when trained on data c
 ollected using RoBERTa itself.
LOCATION:FW26\, Computer Laboratory
END:VEVENT
END:VCALENDAR