Abstract

Abstract:

Current practices of operationalizing annotations in crowdsourced datasets for natural language understanding (NLU) too often assume one single label per item. In this talk, I argue that NLU should investigate disagreement in annotations – human label variation (Plank 2022) – to fully capture human interpretations of language. I investigate how human label variation in natural language inference (NLI) arises, focusing on linguistic phenomena present in the sentences that lead to different interpretations. I also explore two modeling approaches for detecting items with potential disagreement (a 4-way classification with a Complicated label in addition to the three standard NLI labels, and a multilabel classification approach), and evaluate whether these approaches recall the possible interpretations in the data.

Bio:

Marie-Catherine de Marneffe obtained her PhD from Stanford University in 2012. She is an associate professor in Linguistics at The Ohio State University. She also got appointed as a FNRS Research Associate at UCLouvain in 2022. Her research focuses on developing computational linguistic methods that capture what is conveyed by language beyond the literal meaning of the words. In particular she works on “veridicality”: how do people interpret events they read about in the news—do they think such events really happen, did not happen, or are just a possibility? Primarily she wants to ground meanings in corpus data and show how such meanings can drive pragmatic inference. She has also contributed to defining the Stanford Dependencies and the Universal Dependencies representations. Her research has been funded by Google Inc. and the NSF .