Abstract

Crowd IQ: Weighting Votes in Crowdsourcing and Multi-Agent Systems using Item Response Theory

This work introduces a Weighted Majority Voting (WMV) approach for boosting the performance of crowds and multi-agent systems. Our approach relies on the estimation of performance of individual agents based on their ability modelled using Item Response Theory (IRT), a core theory in modern psychological assessment. We provide a brief introduction to IRT and derive the formula defining the weight of individual agents based on their ability and parameters of the task. Using simulated and empirical samples, we show that WMV (1) offers a significant boost in performance across tasks and crowds of different size, (2) is not substantially affected by not knowing the true task parameters-a common scenario in the real-life setting, (3) significantly increases the chances of the crowd to outperform their smartest member, and (4) boosts the marginal contributions of additional agents. Also, using the example of general intelligence, we show that an agent’s ability can be estimated with appropriate accuracy using a relatively short screening test. We conclude with a discussion of the advantages of WMV and suggest that implementing ability- rather than performance-based metrics can offer great advantages in the context of crowdsourcing and multi-agent platforms.