Truth inference for crowdsourcing via bias-variance fusion graph embedding

Abstract

Crowdsourcing has served as an effective method for data collection and labeling. However, the varying abilities of individual workers lead to inconsistent quality in the collected data. Existing truth-inference methods always assign initial reliability values to workers and refine them iteratively. This methodology for estimating worker reliability fails to accurately assess worker reliability and consequently reduces the accuracy of truth inference. In this article, we explore a novel strategy to measure worker reliability by estimating the label quality of each worker for individual instances. Specifically, we introduce two metrics—bias and variance—to quantify, respectively, the discrepancy between a worker’s labels and those of other workers on similar instances (bias), and the inconsistency in a worker’s own labels across different instances (variance). These components are integrated to compute the label quality of each worker for inference instances, enabling accurate assessment of their actual labeling capability and determination their reliability. Building upon this, we propose Graph Embedding for Truth Inference with Worker Reliability (GETI-WR), a bias-variance fusion graph embedding method for crowdsourced truth inference. GETI-WR leverages the obtained worker reliability to map the "task-worker-label" graph into a low-dimensional continuous vector space. Then we use a graph neural network to transform the crowdsourcing problem into a graph-node prediction task to enhance truth-discovery efficiency. Experimental results on two real-world datasets demonstrate that compared with six state-of-the-art baseline methods, our method achieves superior performance in terms of both accuracy and F1-score.