BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations
  for Natural Language Tasks -  Diarmuid Ó Séaghdha (Computer Laboratory)
DTSTART:20081105T130000Z
DTEND:20081105T140000Z
UID:TALK14794@talks.cam.ac.uk
CONTACT:Diarmuid Ó Séaghdha
DESCRIPTION:I'll be presenting and discussing the following paper from EMN
 LP 2008:\n\nRion Snow\, Brendan O'Connor\, Daniel Jurafsky and Andrew Y. N
 g. "Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for
  Natural Language Tasks":http://ai.stanford.edu/~rion/papers/amt_emnlp08.p
 df\n. Proceedings of EMNLP 2008.\n\n*Abstract:*\n\nHuman linguistic annota
 tion is crucial for many natural language processing tasks but can be expe
 nsive and time-consuming. We explore the use of Amazon’s Mechanical Turk
  system\, a significantly cheaper and faster\nmethod for collecting annot
 ations from a broad base of paid non-expert contributors over the Web. We 
 investigate five tasks: affect recognition\, word similarity\, recognizin
 g textual entailment\, event temporal ordering\, and word sense disambigua
 tion. For all five\, we show high agreement between Mechani-\ncal Turk no
 n-expert annotations and existing gold standard labels provided by expert 
 labelers. For the task of affect recognition\, we also show that using non
 -expert labels for training machine learning algorithms can be as effectiv
 e as using gold standard annotations from experts. We propose a technique 
 for bias\ncorrection that significantly improves annotation quality on tw
 o tasks. We conclude that many large labeling tasks can be effectively des
 igned and carried out in this method at a fraction of the usual expense.
LOCATION:GS15\, Computer Laboratory
END:VEVENT
END:VCALENDAR