Abstract

Self-attention architectures allow models to dynamically decide which areas of the input should receive more focus. During the construction of text representations, attention weights also provide a way of quantifying the importance of different input areas. In this talk, we investigate how attention mechanisms can be turned into sequence labelers, opening up some new and interesting applications. These networks learn to predict labels for individual tokens, based only on sentence-level supervision, even without having seen any examples of sequence labeling. In addition, optimizing on the token level explicitly teaches the model where it should be focusing, leading to improvements in text classification. We will also discuss experiments with learning directly from the human cognitive signal, guiding the models to internally behave more like their users. The resulting architectures for text classification and sequence labeling are more accurate, more interpretable and make decisions in more predictable ways.