BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:In-Context Learning - Yashar Ahmadian\; Nandini Shiralkar
DTSTART:20250415T140000Z
DTEND:20250415T153000Z
UID:TALK230212@talks.cam.ac.uk
CONTACT:124819
DESCRIPTION:In-context learning is an emergent capability of large languag
 e models (LLMs) trained via next-token prediction. It refers to the LLMs
 ’ ability to learn new tasks and associations (rules\, patterns\, functi
 ons\, etc)\, without changes in their weights\, based on (often few) examp
 les provided in their active context window. We will mention some examples
  of in-context learning of both natural language and numerical tasks by LL
 Ms\, and quickly review work on their "mechanistic interpretability":https
 ://transformer-circuits.pub/2021/framework/index.html. In particular\, we 
 will present a study linking ICL capabilities of transformer LLMs to the e
 mergence of so-called "induction heads":https://transformer-circuits.pub/2
 022/in-context-learning-and-induction-heads/index.html during their (pre)t
 raining. We will then present a "paper":https://proceedings.neurips.cc/pap
 er_files/paper/2024/file/0ba385c3ea3bb417ac6d6a33e24411bc-Paper-Conference
 .pdf which reveals striking parallels between induction heads in LLMs and 
 the Contextual Maintenance and Retrieval (CMR) model of human episodic mem
 ory. Both exhibit similar behavioural patterns (temporal contiguity and fo
 rward asymmetry)\, converge on nearly identical parameter values\, and use
  functionally equivalent computational mechanisms. This convergence betwee
 n artificial and biological systems offers valuable insights into both LLM
  interpretability and the computational principles underlying sequential m
 emory processing in humans.\n \nPapers:\nhttps://transformer-circuits.pub/
 2021/framework/index.html\nhttps://transformer-circuits.pub/2022/in-contex
 t-learning-and-induction-heads/index.html\nhttps://proceedings.neurips.cc/
 paper_files/paper/2024/file/0ba385c3ea3bb417ac6d6a33e24411bc-Paper-Confere
 nce.pdf
LOCATION:CBL Seminar Room\, Engineering Department\, 4th floor Baker build
 ing
END:VEVENT
END:VCALENDAR