BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:LLMs\, Implicit Bayesian inference and compositional Generalizatio
 n - Szilvia Ujvary (University of Cambridge)
DTSTART:20250620T110000Z
DTEND:20250620T120000Z
UID:TALK231436@talks.cam.ac.uk
CONTACT:Suchir Salhan
DESCRIPTION:**Abstract**\n\nApparently rational behaviors of autoregressiv
 e LLMs\, such as in-context learning\, have been attributed to implicit Ba
 yesian inference: since training data is best explained as a mixture\, the
  optimal next-token-predictor learns to implicitly infer latent concepts a
 nd completes prompts consistently with Bayesian inference. Although it is 
 optimal in-distribution\, Bayesian inference is generally suboptimal on ou
 t-of-distribution prompts due to model misspecification. As model behavior
  on OOD prompts is only weakly constrained by pretraining\, it is not guar
 anteed that Bayesian behavior is extrapolated OOD.  In this talk\, we inve
 stigate with small-scale experiments the degree to which Bayesian inferenc
 e remains a good model of LM behavior on OOD prompts. We first review rela
 ted approaches from the literature. Then\, focusing on small-scale composi
 tional tasks  - learning rules of formal languages - we show that Transfor
 mers can solve harder tasks than trained on\, even in settings where the B
 ayes posterior is undefined. We highlight the role of task compositionalit
 y as a useful inductive bias in enabling models to learn more than the tra
 ining data.\n\n\n**Speaker Biography**\nSzilvia is a second-year PhD stude
 nt working with Professor Ferenc Huszár. Szilvia's research focuses on ex
 plaining emergent abilities of LLMs\, such as in‑context learning and ou
 t‑of‑distribution generalisation\, as well as related foundational que
 stions in (algorithmic) information theory.
LOCATION:ONLINE ONLY. Here is the Zoom link: https://cam-ac-uk.zoom.us/j/4
 751389294?pwd=Z2ZOSDk0eG1wZldVWG1GVVhrTzFIZz09
END:VEVENT
END:VCALENDAR