BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:Overcoming catastrophic forgetting and enabling forward transfer i
 n Continual Learning: A Sparsity Approach - Jonathan Richard Schwarz\, Sen
 ior Research Scientist\, DeepMind
DTSTART:20221020T140000Z
DTEND:20221020T150000Z
UID:TALK183077@talks.cam.ac.uk
CONTACT:104848
DESCRIPTION:As machine learning systems are being applied throughout the s
 ciences and technologies impacting our lives in diverse ways\, the need fo
 r techniques to accumulate skills\, update knowledge and rapidly adapt to 
 novel scenarios is becoming more evident. This ability to learn from a lon
 g sequence of experience rather than fixed data sets\, termed Continual Le
 arning\, is critical to the next generation of ML Systems.\n\nIn this talk
 \, I will first provide a definition of the Continual Learning problem and
  its fundamental desiderata along with a review of the principal solution 
 approaches emerging from the literature in recent years. I will then argue
  that while great progress has been made\, too often have existing approac
 hes over optimised a single objective to the detriment of others. To overc
 ome this problem\, I will then argue that sparsity in its various forms ha
 s recently surfaced as a powerful algorithmic principle that allows the jo
 int optimisation of all desiderata with little interference. \n\nI will su
 pport this view by discussing three approaches spanning various forms of S
 parsity. Firstly\, a discussion of how Sparse Gaussian Processes enable ef
 ficient and principled example selection in Rehearsal-based Continual lear
 ning. Secondly\, the proposal of a simple weight reparameterisation scheme
  for Neural networks\, leading to inherently sparse solutions resulting in
  Continual Learning systems immune to  forgetting. Finally\, a method at t
 he intersection of Continual and Meta-Learning optimised for fast forward 
 transfer with minimal forgetting through sparse Gradient Descent. I conclu
 de with a discussion around directions for future work.
LOCATION:SS03
END:VEVENT
END:VCALENDAR