BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:Learning linear models in-context with transformers - Spencer Frei
 \, UC Davis
DTSTART:20231025T100000Z
DTEND:20231025T113000Z
UID:TALK207712@talks.cam.ac.uk
CONTACT:Isaac Reid
DESCRIPTION:Attention-based neural network sequence models such as transfo
 rmers have the capacity to act as supervised learning algorithms: They can
  take as input a sequence of labeled examples and output predictions for u
 nlabeled test examples.  Indeed\, recent work by Garg et al. has shown tha
 t when training GPT2 architectures over random instances of linear regress
 ion problems\, these models' predictions mimic those of ordinary least squ
 ares.  Towards understanding the mechanisms underlying this phenomenon\, w
 e investigate the dynamics of in-context learning of linear predictors for
  a transformer with a single linear self-attention layer trained by gradie
 nt flow.  We show that despite the non-convexity of the underlying optimiz
 ation problem\, gradient flow with a random initialization finds a global 
 minimum of the objective function.  Moreover\, when given a prompt of labe
 led examples from a new linear prediction task\, the trained transformer a
 chieves small prediction error on unlabeled test examples.  We further cha
 racterize the behavior of the trained transformer under distribution shift
 s.  \n\nBio:\nSpencer Frei is an Assistant Professor of Statistics at UC D
 avis.  His research is on the foundations of deep learning\, including top
 ics related to benign overfitting\, implicit regularization\, and large la
 nguage models.  Prior to joining UC Davis he was a postdoctoral fellow at 
 UC Berkeley hosted by Peter Bartlett and Bin Yu.  He was a co-organizer of
  the 2022 Deep Learning Theory Workshop and Summer School at the Simons In
 stitute for the Theory of Computing.  He received his Ph.D in Statistics f
 rom UCLA in 2021 under the co-supervision of Quanquan Gu and Ying Nian Wu.
LOCATION:Cambridge University Engineering Department\, CBL Seminar room BE
 4-38.
END:VEVENT
END:VCALENDAR