BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:Evaluation Metrics and Learning to Rank for Information Retrieval 
 - Emine Yilmaz - University College London
DTSTART:20140122T140000Z
DTEND:20140122T150000Z
UID:TALK49750@talks.cam.ac.uk
CONTACT:David Greaves
DESCRIPTION:Most current information retrieval systems are machine learnin
 g algorithms that are designed to optimize for evaluation metrics measurin
 g user satisfaction\, a process referred to as learning to rank. Two impor
 tant problems are raised during the learning to rank process: (1) Which ev
 aluation metric should be used as the objective in optimization?\, and (2)
  How to reduce the large number of judgments needed to create the training
  data for learning to rank?\n\nIn the first half of this talk\, I will fir
 st focus on the effect of evaluation metrics used as objectives in learnin
 g to rank. I will first show that in contrast to the common belief\, the t
 arget metric used in optimization is not necessarily the metric that evalu
 ates user satisfaction. I will then describe an information theoretic fram
 ework that can be used to analyze the informativeness of evaluation metric
 s and show that more informative metrics should be used as objectives duri
 ng learning to rank\, independent of the measure that best captures user s
 atisfaction.\n\nIn the second half of the talk\, I will focus on technique
 s that can be used to reduce the number of judgments needed for training a
 nd evaluation of retrieval systems. I will describe a method based on samp
 ling and statistical inference that can be used to reduce the number of ju
 dgments needed for training and evaluation of retrieval systems by 90%\, e
 nabling commercial search engine companies save millions of dollars and re
 search laboratories build their own search engines. The sampling based met
 hods I will describe have been adopted by the National Institute of Standa
 rds and Technology (NIST)\, and are used as standard methods by venues suc
 h as TREC (Text REtrieval Conference)\, CLEF (Cross Language Evaluation Fo
 rum) and INEX (Initiative for Evaluation of XML Retrieval).
LOCATION:Lecture Theatre 1\, Computer Laboratory
END:VEVENT
END:VCALENDAR