BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:On Over-fitting in Model Selection and Subsequent Selection Bias i
 n Performance Evaluation - DR. Tom Minka (Microsoft Research Cambridge)
DTSTART:20110519T130000Z
DTEND:20110519T143000Z
UID:TALK31405@talks.cam.ac.uk
CONTACT:Konstantina Palla
DESCRIPTION:This reading group meeting will discuss the following paper:\n
 \n* "On Over-fitting in Model Selection and Subsequent Selection Bias in P
 erformance Evaluation"\n* by Gavin C. Cawley and Nicola L. C. Talbot\n* Jo
 urnal of Machine Learning Research 11 (2010) 2079-2107\n* http://jmlr.csai
 l.mit.edu/papers/v11/cawley10a.html\n\nModel selection strategies for mach
 ine learning algorithms typically involve the numerical optimisation of an
  appropriate model selection criterion\, often based on an estimator of ge
 neralisation performance\, such as k-fold cross-validation. The error of s
 uch an estimator can be broken down into bias and variance components. Whi
 le unbiasedness is often cited as a beneficial quality of a model selectio
 n criterion\, we demonstrate that a low variance is at least as important\
 , as a non-negligible variance introduces the potential for over-fitting i
 n model selection as well as in training the model. While this observation
  is in hindsight perhaps rather obvious\, the degradation in performance d
 ue to over-fitting the model selection criterion can be surprisingly large
 \, an observation that appears to have received little attention in the ma
 chine learning literature to date. In this paper\, we show that the effect
 s of this form of over-fitting are often of comparable magnitude to differ
 ences in performance between learning algorithms\, and thus cannot be igno
 red in empirical evaluation. Furthermore\, we show that some common perfor
 mance evaluation practices are susceptible to a form of selection bias as 
 a result of this form of over-fitting and hence are unreliable. We discuss
  methods to avoid over-fitting in model selection and subsequent selection
  bias in performance evaluation\, which we hope will be incorporated into 
 best practice. While this study concentrates on cross-validation based mod
 el selection\, the findings are quite general and apply to any model selec
 tion practice involving the optimisation of a model selection criterion ev
 aluated over a finite sample of data\, including maximisation of the Bayes
 ian evidence and optimisation of performance bounds.
LOCATION:Engineering Department\, CBL Room 438
END:VEVENT
END:VCALENDAR