BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:The potential of synthetic data for more informative evaluation in
  Visual Question Answering - Alexander Kuhnle\, NLIP\, University of Cambr
 idge
DTSTART:20180518T110000Z
DTEND:20180518T120000Z
UID:TALK104641@talks.cam.ac.uk
CONTACT:Andrew Caines
DESCRIPTION:Visual Question Answering (VQA) combines language and scene un
 derstanding in a straightforward task. However\, the popular VQA Dataset e
 xhibits characteristics which make it easy for even a simple system to ach
 ieve surprisingly good performance. At the same time\, sophisticated model
  improvements are often barely\, if at all\, reflected in corresponding pe
 rformance gains. I will discuss various reasons for that\, and why this ty
 pe of monolithic real-world dataset as *sole* benchmark thus might be a de
 ad end for VQA. Various synthetic abstract VQA datasets have recently been
  published\, to help overcome this problem. I will introduce "ShapeWorld"\
 , our framework for automatically generating abstract multimodal data\, di
 scuss some of the design choices of the generation system\, and contrast i
 t with related synthetic datasets. Central to our system is an evaluation 
 methodology akin to "unit-testing"\, in contrast to real-world datasets wh
 ich serve more as application benchmarks. Finally\, I will present some ex
 perimental results\, focusing on quantifier statements\, which illustrate 
 this approach and how it enables more targeted and detailed analysis of de
 ep learning models.\n
LOCATION:FW26\, Computer Laboratory
END:VEVENT
END:VCALENDAR