R. Bellio

Accurate agreement estimation in crowdsourced relevance assessments via statistical models based on the beta distribution

Data crowdsourcing is often used to collect relevance assessments in information retrieval tasks. However, estimating the agreement among crowdworkers is nontrivial, since data is sparse and often collected on a variety of rating scales. We consider a model‐based agreement measure defined as a one‐to‐one transformati...