BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:SummaryMixing: A Linear-Time Attention Alternative - Shucong Zhang
 \, Samsung AI Center
DTSTART:20241021T110000Z
DTEND:20241021T120000Z
UID:TALK222598@talks.cam.ac.uk
CONTACT:Simon Webster McKnight
DESCRIPTION:Modern speech processing systems rely on self-attention. Unfor
 tunately\, self-attention takes quadratic time in the length of the speech
  utterance\, causing inference and training on long sequences to be slower
  and consume more memory. Though cheaper alternatives to self-attention fo
 r speech recognition have been developed\, they degrade performance. We pr
 opose a novel linear-time alternative to self-attention that\, for the fir
 st time\, does reach better accuracy. Our model\, SummaryMixing\, computes
  a mean over the whole utterance and feeds this summary back to each time 
 step.Experiments are performed in three vital scenarios: an encoder-decode
 r offline model\; an online streaming Transducer model\; and a self-superv
 ised model. In all three scenarios\, SummaryMixing gives equal or better a
 ccuracy than self-attention\, at lower cost.
LOCATION:Hybrid: JDB Teaching Room\, Engineering Department or Zoom: https
 ://cam-ac-uk.zoom.us/j/87165608116?pwd=llCLiWvBAfOR7RtealbVtVbjXruh3O.1
END:VEVENT
END:VCALENDAR