Abstract

As machine learning systems are being applied throughout the sciences and technologies impacting our lives in diverse ways, the need for techniques to accumulate skills, update knowledge and rapidly adapt to novel scenarios is becoming more evident. This ability to learn from a long sequence of experience rather than fixed data sets, termed Continual Learning, is critical to the next generation of ML Systems.

In this talk, I will first provide a definition of the Continual Learning problem and its fundamental desiderata along with a review of the principal solution approaches emerging from the literature in recent years. I will then argue that while great progress has been made, too often have existing approaches over optimised a single objective to the detriment of others. To overcome this problem, I will then argue that sparsity in its various forms has recently surfaced as a powerful algorithmic principle that allows the joint optimisation of all desiderata with little interference.

I will support this view by discussing three approaches spanning various forms of Sparsity. Firstly, a discussion of how Sparse Gaussian Processes enable efficient and principled example selection in Rehearsal-based Continual learning. Secondly, the proposal of a simple weight reparameterisation scheme for Neural networks, leading to inherently sparse solutions resulting in Continual Learning systems immune to forgetting. Finally, a method at the intersection of Continual and Meta-Learning optimised for fast forward transfer with minimal forgetting through sparse Gradient Descent. I conclude with a discussion around directions for future work.