Abstract

Effective virtual assistants have to guide users through long and complex tasks, be engaging, and help solve unpredictable problems along the way. This talk presents GRILL Bot, a multi-modal task-oriented assistant, as the winning system in the 2022 Amazon Alexa Prize TaskBot challenge. GRILL Bot personalizes assistance in cooking, DIY , and crafts by leveraging TaskGraphs, a novel task representation. TaskGrahps unify steps, requirements, and curated domain knowledge into dynamic graphs enabling detailed contextual explanations and adaptable task execution. Further augmentations with segmented videos and linked step images allow full use of the multi-modal experience. We will explore the critical roles language models play in GRILL Bot both in offline task augmentations like contextual jokes and in online settings with knowledge-grounded question answering and a semantic parsing approach to dialogue management. We will finally discuss the lessons learned from a full year of real-world users and showcase GRILL Bot in action.