Abstract

Abstract: The first half of this talk will describe the feature learning problem in deep learning optimization, its statistical consequences, and an approach to proving general theorems with a heavy reliance on normalization layers, which are common to all modern architectures but typically treated as an analytic nuisance. Theorems will cover two settings: concrete results for shallow networks, and abstract template theorems for general architectures.

The second half will survey proof techniques. The two key ingredients are a careful new mirror descent lemma, derived from the work of Chizat and Bach, and a new characterization of common layer types called lower homogeneity.

Joint work with Danny Son.

Bio: Matus Telgarsky is an assistant professor in the Courant Institute, NYU , specializing in deep learning theory. He was fortunate to receive a PhD at UCSD under Sanjoy Dasgupta. Other highlights include: co-founding, in 2017, the Midwest ML Symposium (MMLS) with Po-Ling Loh; receiving a 2018 NSF CAREER award; and organizing two Simons Institute programs, one on deep learning theory (summer 2019), and one on generalization (fall 2024, again with Po-Ling Loh); having lots of good friends and too many fun things to do.