University of Cambridge > Talks.cam > Computational Neuroscience > In-Context Learning

  Browse A-Z

Log in

Googledetails Microsoftdetails

🔐 Log In

Information on

Subscribing to talksdetails Finding a talkdetails Adding a talkdetails Disseminating talksdetails Help and Documentationdetails

ℹ️ Information

In-Context Learning

Download to your calendar using vCal

If you have a question about this talk, please contact .

In-context learning is an emergent capability of large language models (LLMs) trained via next-token prediction. It refers to the LLMs’ ability to learn new tasks and associations (rules, patterns, functions, etc), without changes in their weights, based on (often few) examples provided in their active context window. We will mention some examples of in-context learning of both natural language and numerical tasks by LLMs, and quickly review work on their mechanistic interpretability. In particular, we will present a study linking ICL capabilities of transformer LLMs to the emergence of so-called induction heads during their (pre)training. We will then present a paper which reveals striking parallels between induction heads in LLMs and the Contextual Maintenance and Retrieval (CMR) model of human episodic memory. Both exhibit similar behavioural patterns (temporal contiguity and forward asymmetry), converge on nearly identical parameter values, and use functionally equivalent computational mechanisms. This convergence between artificial and biological systems offers valuable insights into both LLM interpretability and the computational principles underlying sequential memory processing in humans.

Papers: https://transformer-circuits.pub/2021/framework/index.html https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html https://proceedings.neurips.cc/paper_files/paper/2024/file/0ba385c3ea3bb417ac6d6a33e24411bc-Paper-Conference.pdf

This talk is part of the Computational Neuroscience series.

This talk is included in these lists:

Note that ex-directory lists are not shown.

Other lists

Institute of Criminology, B3, Sidgwick Site, Faculty of History, Sidgwick Site Room 713, Link Room, Cavendish Laboratory

Other talks

title abstract tbc Online Causal Inference Seminar Teams Announced Title TBC Harnessing Radical Reactivity for Molecular Innovation Novel neuroimaging methods in Lewy body dementia
 

© 2006-2025 Talks.cam, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity