BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:Random Features for Kernel Approximation - Isaac Reid (University 
 of Cambridge)
DTSTART:20230308T110000Z
DTEND:20230308T123000Z
UID:TALK198160@talks.cam.ac.uk
CONTACT:James Allingham
DESCRIPTION:Though ubiquitous and mathematically elegant\, kernel methods 
 notoriously suffer from poor scalability as dataset size grows on account 
 of the need to store and invert the Gram matrix. This has motivated a numb
 er of kernel approximation techniques. Chief among them are random feature
 s\, which construct low-rank decompositions to the Gram matrix via Monte C
 arlo methods. We begin by discussing Rahimi and Recht’s seminal paper on
  Random Fourier Features\, which approximates stationary kernels with a ra
 ndomised sum of sinusoids. We briefly draw parallels to the celebrated Joh
 nson-Lindenstrauss transform\, before discussing how Orthogonal Random Fea
 tures enjoy better convergence. We demonstrate the effectiveness of these 
 techniques for approximating attention in Transformers. Finally – if you
  will humour me – we will briefly discuss how carefully induced correlat
 ions between random features can further improve the quality of kernel app
 roximation\, describing the recently-introduced class of Simplex Random Fe
 atures.  \n\nPapers: \n\nRahimi\, A. and Recht\, B. (2007). Random feature
 s for large-scale kernel machines. Advances in neural information processi
 ng systems\, 20. \n\nJohnson\, W. B. (1984). Extensions of Lipschitz mappi
 ngs into a Hilbert space. Contemp. Math.\, 26:189–206.\n\nYu\, F. X. X.\
 , Suresh\, A. T.\, Choromanski\, K. M.\, Holtmann-Rice\, D. N.\, and Kumar
 \, S. (2016). Orthogonal random features. Advances in neural information p
 rocessing systems\, 29.\n\nChoromanski\, K.\, Likhosherstov\, V.\, Dohan\,
  D.\, Song\, X.\, Gane\, A.\, Sarlos\, T.\, Hawkins\, P.\, Davis\, J.\, Mo
 hiuddin\, A.\, Kaiser\, L.\, et al. (2020). Rethinking attention with perf
 ormers. International Conference on Learning Representations\, 9. \n\nReid
 \, I.\, Choromanski\, K.\, Likhosherstov\, V.\, and Weller\, A. (2023). Si
 mplex random features. arXiv preprint arXiv:2301.13856
LOCATION:Cambridge University Engineering Department\, CBL Seminar room BE
 4-38.
END:VEVENT
END:VCALENDAR