BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:The Cambridge ELLIS Unit Seminar Series - Gabriel Synnaeve - Speak
 er to be confirmed
DTSTART:20240725T130000Z
DTEND:20240725T140000Z
UID:TALK219154@talks.cam.ac.uk
CONTACT:123034
DESCRIPTION:The Cambridge ELLIS Unit Seminar Series holds talks by leading
  researchers in the area of machine learning and AI. Our next speaker for 
 July 2024 will be Gabriel Synnaeve. Details of his talk can be found below
 . \n\nTitle: “Multi-token Prediction and Exploring LM Losses"\n\nAbstarc
 t: \nLarge language models such as GPT and Llama are trained with a next-t
 oken prediction loss. In this work\, we suggest that training language mod
 els to predict multiple future tokens at once results in higher sample eff
 iciency. More specifically\, at each position in the training corpus\, we 
 ask the model to predict the following n tokens using n independent output
  heads\, operating on top of a shared model trunk. Considering multi-token
  prediction as an auxiliary training task\, we measure improved downstream
  capabilities with no overhead in training time for both code and natural 
 language models. The method is increasingly useful for larger model sizes\
 , and keeps its appeal when training for multiple epochs. Gains are especi
 ally pronounced on generative benchmarks like coding\, where our models co
 nsistently outperform strong baselines by several percentage points. Our 1
 3B parameter models solves 12 % more problems on HumanEval and 17 % more o
 n MBPP than comparable next-token models. Experiments on small algorithmic
  tasks demonstrate that multi-token prediction is favorable for the develo
 pment of induction heads and algorithmic reasoning capabilities. As an add
 itional benefit\, models trained with 4-token prediction are up to 3 times
  faster at inference\, even with large batch sizes.\n\nhttps://cam-ac-uk.z
 oom.us/j/85608707597?pwd=Ga0BEodRBJpHkKwvKNdCjabGhBNUYU.1\n
LOCATION:https://cam-ac-uk.zoom.us/j/85608707597?pwd=Ga0BEodRBJpHkKwvKNdCj
 abGhBNUYU.1
END:VEVENT
END:VCALENDAR