BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:Learning and testing compositionality - Elia Bruni\, University of
  Pompeu Fabra (Barcelona\, Spain)
DTSTART:20190124T110000Z
DTEND:20190124T120000Z
UID:TALK118552@talks.cam.ac.uk
CONTACT:Edoardo Maria Ponti
DESCRIPTION:While sequence-to-sequence (seq2seq) models have shown remarka
 ble generalisation power across several natural language tasks\, their con
 struct of solutions is argued to be less compositional than human-like gen
 eralisation. In this talk\, I will discuss our attempts to narrow this gap
 .\n\nIn the first part of the talk\, I will introduce the notion of compos
 itionality and explain why it is crucial for artificial learners to master
  it if they want to talk and think like people. I will then present three 
 different strategies we followed to bias seq2seq models towards more compo
 sitional solutions.\n\nFirst\, I will talk about Attentive Guidance (AG)\,
  a new mechanism to direct a seq2seq model equipped with attention to find
  more compositional solutions. Models trained with AG come up with solutio
 ns that\, in some cases\, fit the training and testing distributions\nequa
 lly well. \n\nWhile AG is effective\, it has the problem that needs an ext
 ra supervision signal. As a remedy\, I will present sequence-to-attention\
 , a new architecture that we specifically designed to exploit attention to
  find compositional patterns in the input without the need of extra-superv
 ision. The solutions found by the model are highly interpretable\, allowin
 g easy analysis of both the types of solutions that are found and potentia
 l causes for mistakes. \n\nLastly\, I will present some preliminary result
 s of a second architecture where we are trying to disentangle content- and
  position-based representations in the attention mechanism of a seq2seq. T
 his again helps with interpretability but also with extrapolating to longe
 r sequences than the ones seen by the model during training. Furthermore\,
  it gives us the chance to design a more human-like version of (positional
 ) attention.\n\nIn the second part of the talk\, I will argue that current
 ly\,\nit is difficult to test for compositionality in neural networks. I w
 ill then present our compositional manifesto\, a new battery of tests to a
 ssess the compositional abilities in seq2seq models. In particular\, I wil
 l introduce five tests: Localism\, Substitutivity\, Productivity\, Systema
 ticity and Overgeneralisation. I will then "test the tests" using three in
 stances of a seq2seq model: a recurrent-seq2seq\, a convolutional-seq2seq\
 , and a Transformer network. \n\nTo conclude\, I will highlight new resear
 ch directions relating to compositional learning where I aim to ground the
  learners in the visual world.
LOCATION:Faculty of English\, Room SR24
END:VEVENT
END:VCALENDAR