An optimization approach for worker selection in crowdsourcing systems

Highlights

• Workers with different qualities are selected from the perspective of minimum cost.

• Optimization models are developed for static and dynamic crowdsourcing systems.

• Extensive simulation studies are given to validate the proposed methods.

Abstract

Crowdsourcing has attracted considerable attention in recent years. A large amount of labelled data can be obtained efficiently and cheaply from the crowdsourcing platform. Therefore, providing a complete, feasible and efficient paradigm for marking unlabelled data in crowdsourcing is in demand. Since the quality of crowd workers directly influences the quality of the labelled data, an optimization model subjecting to quality constraints is built in this paper, aiming to minimize cost function designed based on worker quality. Cost-controlled and quality-assured worker selection algorithms are proposed to solve crowd-labelling problems for static and dynamic crowdsourcing systems separately. Numerical examples based on collected data validate the feasibility of the proposed algorithms. Through extensive simulations, it is demonstrated the proposed method can keep a trade-off between the total cost and the accuracy of label inference which is assured via worker selection. In addition, the proposed approach can also adjust the parameters flexibly according to requesters’ demand for the accuracy or budget requirements, which shows its practicality.

Introduction

With the development of machine learning especially supervised learning algorithms, there is a surge in demand for extensive labelled data used to train a model for making predictions on new data, which supports applications including image classification, object detection, semantic-based information retrieval, and natural language processing (Tao et al., 2020). However, labelled data are obtained by designing sophisticated algorithms at the expense of considerable time or spending a lot of money consulting domain experts or well-trained workers, which is not economical and efficient. In this situation, crowdsourcing provides a time-saving and high-efficiency approach to obtaining substantial amounts of labels by hiring crowd workers. Crowdsourcing has also been recognized as an effective way to complete tasks that are easy for people but difficult for computers (Allahbakhsh et al., 2013). Crowdsourcing was first proposed by Howe in 2006 (Howe, 2006) as a further development of outsourcing to validate, modify and/or improve a company’s value. (Ghezzi et al., 2018). Crowdsourcing is a distributed problem-solving, production model based on an internet platform that can assist organizations to complete one-time or rarely occurring tasks that require large numbers of workers. Depending on the tasks, the required workers may need to have relevant skills or may not, however, by using a certain aggregation mechanism in many tasks, even unskilled participants can eventually get good results. Some researchers have harnessed the wisdom of ordinary crowds to predict the outcome of sporting events and have achieved good results (Alasdair & James, 2018). Crowdsourcing has increased productivity in both public and private organizations as it has allowed them to leverage an external (skilled) workforce in addition to their core workforce, reduce training costs, and improve core and support processes. For example, rather than designing sophisticated algorithms or spending a lot of money to consult experts, crowdsourcing workers have been used to label pictures selected to train image tagging algorithms (Gao et al., 2013). Many crowdsourcing platforms have now been established: Amazon Mechanical Turk (AMT) (mturk. com), Galaxy Zoo (galaxyzoo. org), Threadless (threadless. com), and Kaggle (kaggle. com). In a crowdsourcing system, users can post any Human Intelligence Tasks (HITs) that computers are currently unable or difficult to perform, e.g., annotating images of cats versus images of non-cats as shown in Fig. 1. Crowdsourcing makes it possible for not just one but many independent, relatively inexpensive workers (experts and non-experts) to offer their opinions (labels) on the HITs and determine a solution by aggregating these crowd opinions (Wang & Zhou, 2015).

Quality control in crowdsourcing involves two main elements: worker quality and process control quality (Allahbakhsh et al., 2013). The former is achieved via proper worker selection and the latter is realized via redundant work which means allocating the same task to multiple workers. Factors influencing worker quality assessment include basic worker information such as age, gender, credentials such as skills and certification, and experience such as badges, reliability and reputation. The factors above always give effect to the worker’s performance in golden tasks but also history tasks. Labels collected from workers in crowdsourcing vary in quality due to their individual differences including personal preferences and cognitive abilities and dynamics between the workers. Low-quality labelled data may compromise practical applications. Therefore, quality control has been the focus of a great deal of research, intending to quantify the qualifying workers and tasks (Tong et al., 2020). Dang et al. (2015) proposed a general worker quality method that had specific variables for workers and potential problems that could measure quality without wasting resources. Zhang et al. (2018) proposed several algorithms to simultaneously manage quality and reduce crowdsourcing uncertainty. Fang et al. (2018) designed an inference algorithm that incorporated context information and introduced an iterative method to improve quality and attract higher-quality workers. Goel and Faltings (2018) proposed a transitivity mechanism based on golden tasks to incentive workers and measure trust. Yuen et al. (2021) developed a factor analysis model which considers performance history. Zhang and Su (2019) used the combined fuzzy DEMATEL method and TOPSIS method to assess participants in knowledge-intensive crowdsourcing.

Many of these approaches eliminate workers by means of quality control, however, costs in crowdsourcing process are generally ignored. By assigning multiple workers to a single task, higher accuracy of answer estimation may be obtained, but a higher cost may suffer due to hiring more workers. An additional challenge is the question of the trade-off between cost and quality. Some studies have focused on balancing cost and quality. Han and Wu (2017) developed a solution to a minimum cost crowdsourcing problem in a device-to-device network. Adamska et al. (2020) proposed a real-time dynamic filtering algorithm to increase the cost-effectiveness of crowdsourcing. Haruna et al. (2019) proposed an algorithm that used Euclidean distance to solve cost problems at the source and maximize social welfare and chromatic correlation clustering to delete some tasks before the crowdsourcing began to save costs. Shi et al. (2019) proposed a price transfer mechanism to balance the tasks that needed fewer workers or more workers. Two methods have been used to set costs, the first of which has been to have fixed costs for different workers to make it easier to calculate the total costs and to attract a greater number of participants, however, for poorer quality workers, this has been found to result in unnecessary costs. Gao et al. (2013) examined worker quality and fixed costs and proposed linear and non-linear models to effectively control the crowdsourcing costs. Because of the disadvantages associated with the fixed costs, some studies have combined costs and worker quality to ensure greater crowdsourcing reliability. Based on confusion matrices, Hirth et al. (2013) divided crowdsourcing costs into four categories to resolve the quality uncertainties caused by cost and worker anonymity. Karger et al. (2014) designed an algorithm to decide which tasks to assign to which workers (different cost requirements) for a minimum total price. Reliable worker-reputation systems have also been developed to measure worker quality. Moayedikia et al. (2019) proposed an unsupervised approach to estimate worker abilities that considered the possibility that worker quality could change over time, which can overcome the task matching limitations in previous dynamic crowdsourcing system studies of Markov decision making (Dai et al., 2013, Dickerson et al., 2018).

Worker selection, which has also been a common strategy for improving crowdsourcing quality, is the selection of satisfactory workers for the designated tasks, or from another perspective, is a worker elimination problem (Allahbakhsh et al., 2013, Raykar and Yu, 2012). There have been some recent studies on worker selection or task allocation methods in different settings. Yadav et al. (2022) proposed a nondominated sorting genetic algorithm II (NSGA II) based algorithm to find the best set of workers that can perform the task. To dynamically select an optimal candidate set of crowd workers, Wang et al. (2020) proposed an online auction algorithm that combined a multi-attribute auction and a reverse auction that considered trust, privacy sensitivity and other attributes. Zhao et al., 2017a, Zhao et al., 2017b integrated social networks into worker selection and selected trusted workers to complete the tasks through worker positions in the social networks. Wu et al. (2021) considered the interactions between social relationships and crowdsourcing and proposed a worker–task accuracy estimation algorithm based on a graph model to assign tasks to workers. Awwad et al. (2017) proposed an offline learning algorithm that grouped tasks from history into homogeneous clusters and learned the worker features in each cluster to optimize contribution quality. Abououf et al. (2019) proposed a group-based multi-task worker selection model that allocated multiple tasks to workers while maximizing the service quality of the tasks and minimizing completion times. Abdullah et al. (2020) developed a framework with various specifications based on workers’ attributes for the selection of optimal workers for every task, for which a Bayesian Network was employed for the modelling and worker selection and a k-medoids partitioning technique has been used for the task clustering and scheduling. Huang and Ardiansyah (2019) used mixed-integer linear programming to solve the problem of how to select crowdsourcing workers for the last-mile delivery and used heuristic algorithms to handle large-scale problems. Azzam et al. (2018) proposed a stability-based group recruitment system for continuous sensing that employs a genetic algorithm to select groups of participants, and the Shapley value is used to reward selected workers based on their respective contributions.

In prior work, well-designed methods for worker selection or label inference have achieved excellent results. Our work focused on a complete, feasible and efficient paradigm for marking unlabelled data in crowdsourcing, and a cost-controlled and quality-assured worker selection approach is the most important tool to achieve the goal. The contributions of this paper are as follows.

(1) This paper presents a complete worker selection framework for redundantly crowd labelling via keeping a trade–off between the total cost and the accuracy of label inference. Definition of worker quality taking worker’s performance in golden tasks but also history tasks into account is developed. Based on the above, an optimization model subjecting to quality constraints is built with the aim to minimize cost function which is designed based on worker quality.

(2) Two algorithms with optimization models for worker selection applying to static and dynamic crowdsourcing systems are proposed. Their starting points are both the selection of proper workers to assure the accuracy of label inference from a minimum cost perspective. The difference between them mainly manifested in controlling the number of workers who joined the tasks. For static crowdsourcing systems that have a defined number of participants, only one optimization round is needed, which saves time. A dynamic crowdsourcing system, however, decides the number of workers participating in labelling tasks based on the quality control of tasks.

(3) Extensive simulations are given to demonstrate the feasibility and validity of the proposed optimization algorithms.

The remainder of this paper is organized as follows. Section 2 explains the problem and gives the relevant notations. Section 3 presents the new method for measuring worker quality and then builds our cost function. Section 4 develops a minimum cost model and true label algorithm inference for a static crowdsourcing system, for which a numerical example is given to verify its application. Section 5 expands the minimum cost model and the true label inference algorithm from Section 4 to make them applicable to dynamic crowdsourcing systems. Section 6 designs an experiment to validate the proposed algorithms and conducts simulation analyses to offer insights into the behaviour of the theoretical models. Concluding remarks are given in Section 7.