BEGIN:VCALENDAR VERSION:2.0 PRODID:-//Talks.cam//talks.cam.ac.uk// X-WR-CALNAME:Talks.cam BEGIN:VEVENT SUMMARY:"\;What it can create\, it may not understand"\; Studying the Limits of Transformers. - Nouha Dziri\, Allen Institute for AI DTSTART:20240509T100000Z DTEND:20240509T110000Z UID:TALK215491@talks.cam.ac.uk CONTACT:Panagiotis Fytas DESCRIPTION:Transformer large language models (LLMs) have sparked admirati on for their exceptional performance on tasks that demand intricate multi- step reasoning. They only take seconds to produce outputs that would chall enge or exceed the capabilities even of expert humans. Yet\, these models simultaneously show failures on surprisingly trivial problems. This presen ts us with an apparent paradox: how do we reconcile seemingly superhuman c apabilities with the persistence of errors that few humans would make? Are these errors incidental\, or do they signal more substantial limitations? \nIn an attempt to demystify Transformers\, in this talk\, I will discuss the limits of LLMs across three different compositional tasks. Our findin gs show that although LLMs can outperform humans in generation\, they cons istently fall short of human capabilities in measures of understanding\, s howing weaker correlation between generation and understanding performance \, and more brittleness to adversarial inputs. We further show that transf ormers can often solve multi-step compositional problems by reducing multi -step compositional reasoning into linearized subgraph matching\, without necessarily developing systematic problem-solving skills.\nOverall\, our f indings support the hypothesis that models’ generative capability may no t be contingent upon understanding capability\, and call for caution in in terpreting artificial intelligence by analogy to human intelligence. LOCATION:https://cam-ac-uk.zoom.us/j/97599459216?pwd=QTRsOWZCOXRTREVnbTJBd XVpOXFvdz09 END:VEVENT END:VCALENDAR