BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:Crowd IQ: Weighting Votes in Crowdsourcing and Multi-Agent Systems
  using Item Response Theory - Dr. Michal Konsinski
DTSTART:20140429T153000Z
DTEND:20140429T163000Z
UID:TALK52360@talks.cam.ac.uk
CONTACT:Chan Yin Wah Fiona
DESCRIPTION:Crowd IQ: Weighting Votes in Crowdsourcing and Multi-Agent Sys
 tems using Item Response Theory\n \nThis work introduces a Weighted Majori
 ty Voting (WMV) approach for boosting the performance of crowds and multi-
 agent systems. Our approach relies on the estimation of performance of ind
 ividual agents based on their ability modelled using Item Response Theory 
 (IRT)\, a core theory in modern psychological assessment. We provide a bri
 ef introduction to IRT and derive the formula defining the weight of indiv
 idual agents based on their ability and parameters of the task. Using simu
 lated and empirical samples\, we show that WMV (1) offers a significant bo
 ost in performance across tasks and crowds of different size\, (2) is not 
 substantially affected by not knowing the true task parameters-a common sc
 enario in the real-life setting\, (3) significantly increases the chances 
 of the crowd to outperform their smartest member\, and (4) boosts the marg
 inal contributions of additional agents. Also\, using the example of gener
 al intelligence\, we show that an agent's ability can be estimated with ap
 propriate accuracy using a relatively short screening test. We conclude wi
 th a discussion of the advantages of WMV and suggest that implementing abi
 lity- rather than performance-based metrics can offer great advantages in 
 the context of crowdsourcing and multi-agent platforms.
LOCATION:Seminar Room\, Department of Psychology\, Downing Site\, Cambridg
 e
END:VEVENT
END:VCALENDAR