BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:Evaluating Large Language Models as Model Systems for Language - C
 arina Kauf\, MIT Brain and Cognitive Sciences
DTSTART:20240607T130000Z
DTEND:20240607T140000Z
UID:TALK213220@talks.cam.ac.uk
CONTACT:Richard Diehl Martinez
DESCRIPTION:In this talk\, we investigate the potential of Large Language 
 Models to serve as model systems for language. Model systems for language 
 should first and foremost perform the relevant function\, i.e.\, use langu
 age in the right way. In the first part of the talk we investigate this cl
 aim in two ways. First\, we critically look at model evaluation. To invest
 igate model performance\, it is often beneficial to evaluate and compare t
 he models' performance on controlled sentence generation benchmarks. Here\
 , we argue that Masked Language Model performance has been systematically 
 underestimated due to a bias in the most commonly used sentence/word scori
 ng method: Pseudo-log likelihood. We introduce an improved version of the 
 scoring method which mitigates the observed bias. Then we evaluate if LLMs
  use language in a way consistent with humans’ generalized knowledge of 
 common events\, which is tightly linked with their language behavior. Over
 all\, our results show that important aspects of event knowledge naturally
  emerge from distributional linguistic patterns\, but also highlight a gap
  between representations of possible/impossible and likely/unlikely events
 . In the second part of the talk we shift gears and investigate LLMs as mo
 del systems more directly: We leverage artificial neural network language 
 models as computational hypotheses of language processing in the human bra
 in and measure the degree of alignment between the two systems when proces
 sing a variety of language stimuli. We find substantial alignment between 
 the two systems and systematically investigate features that drive the obs
 erved similarity.\n
LOCATION:https://cam-ac-uk.zoom.us/j/88532356932?pwd=IVo8GI7wBssnObu7in3aN
 XBjwufHnJ.1
END:VEVENT
END:VCALENDAR