Accurate agreement estimation in crowdsourced relevance assessments via statistical models based on the beta distribution

Abstract

Data crowdsourcing is often used to collect relevance assessments in information retrieval tasks. However, estimating the agreement among crowdworkers is nontrivial, since data is sparse and often collected on a variety of rating scales. We consider a model‐based agreement measure defined as a one‐to‐one transformation of the precision parameter of a beta distribution. To avoid assumptions about the distribution of item‐specific relevance parameters, we model them as fixed effects and address the resulting incidental parameter problem via modified profile likelihood (MPL) estimation. The proposed estimator is developed for continuous beta ratings and further extended to discrete and ordered beta models. Simulation studies show that the resulting estimators provide accurate inference in sparse settings and outperform standard likelihood‐based approaches. Applications to crowdsourced relevance assessments illustrate the effectiveness of the proposed methods and highlight how discreteness and boundary inflation affect agreement estimation. The methods are implemented in an R package.