Temporal context-aware task recommendation in crowdsourcing systems
Introduction
Crowdsourcing is an idea of outsourcing a task to a large group of networked people in the form of an open call to reduce the production cost [1], [2]. In recent years, crowdsourcing systems attract much attention at present [3], [4], [5], [6]. Some examples of crowdsourcing systems are Amazon Mechanical Turk (or MTurk) [7], CrowdFlower [8], Taskcn [9] and TopCoder [10]. To achieve quality assurance in crowdsourcing systems, a requester has to verify the quality of every answer submitted by workers, and it is very time-consuming. Alternatively, requesters highly rely on redundancy of answers provided by multiple workers with varying expertise, but massive redundancy is very expensive and time-consuming. “If we ask 10 workers to complete the same task, then the cost of crowdsourcing solutions tends to be comparable to the cost of in-house solutions” [11]. Therefore, it is important to investigate how to enable task requesters to verify correct answers on crowdsourcing platforms more easily and effectively. Chilton et al. [12] showed that task workers look mostly at the first page of the most recently posted tasks and the first two pages of the tasks with the most available instances. The worker performance history including workers’ task searching behavior makes it possible to mine workers’ preference on tasks and to provide an indication of worker quality on tasks. Therefore, based on worker performance histories, task recommendation can help to provide a list of preferred tasks to workers in crowdsourcing systems to guarantee the quality of work done. However, workers’ preference on tasks might change from time to time, so it is necessary to consider the temporal change of workers’ preference on tasks when making task recommendations.
In addition, it is important to help workers to find their right tasks as quick as possible and minimize the number of task assignments to achieve a target output quality [13]. Active learning can be applied on task recommendation to help requesters to receive good quality output quicker with lower cost, thus guaranteeing quality in a shorter period of time.
Probabilistic Matrix Factorization (PMF) [14] is the state-of-the-art approach for recommendation systems. A factorization model has to be trained and learned before the model can be applied for prediction. In real-world applications, the performance of a factorization model is highly affected by how the model is updated, and thus dynamic updating a model is very important [15].
In our previous works [16], [17], we proposed a task recommendation framework which performs a factor analysis based on probabilistic matrix factorization (PMF) with which the worker and task latent feature spaces are learned. Our framework adopts an active learning approach to achieve a target output quality with the minimum task assignments. Moreover, our framework uses an online-updating method on model update process to greatly improve the system performance in terms of the running time of model update and the prediction accuracy.
Workers’ preference might change over time, but all the research works in the literature in task recommendation do not consider the temporal change of workers’ preference on tasks. As a result, they cannot make recommendations depending on fresh and novel workers’ preference on tasks.
Our contributions are:
- •
to propose a method for task recommendation by performing unified matrix factorization on worker-task preference matrix, worker-category preference matrix and task-category grouping matrix with which the worker and task latent feature spaces are learned with a special constraint on the time dimension,
- •
to propose an aging scheme for task recommendation by depending on mostly fresh and novel worker preferences in order to reflect the change of workers’ opinions in real world applications,
- •
to propose an active learning method for task recommendation by selecting the most informative task and the most skilful worker in order to guarantee the output quality of the factor analysis model,
- •
to propose an online-updating method for task recommendation in order to reduce the cost of model update,
- •
to propose a generic online-updating method when learning a factor analysis model,
- •
to show that our model is efficient and is scalable to large datasets by means of complexity analysis,
- •
to demonstrate the performance of our task recommendation framework by using real-world datasets,
- •
to study the effects of various parameters on our task recommendation framework by using synthetic datasets.
To the best of our knowledge, our work is the first research work considering temporal factors in task recommendation. The rest of this paper is organized as follows. Section 2 presents the related works. Section 3 presents our proposed time-aware task recommendation framework in crowdsourcing systems. Section 4 presents our proposed active learning method. Section 5 presents our proposed online-updating method. Section 6 describes our experiments. Section 7 concludes with future directions.
Section snippets
Related work
Recommendation systems can model users’ interest and perform interest-based user recommendation. With the exponential growth of information on the Web, social recommendation has attracted much attention [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36]. Social recommendation uses computational techniques (such as machine learning and data mining) on social behavior data collected from blogs and tags, etc. from social media (such as
Time-aware task recommendation framework
We propose a framework for Time-Aware TAsk RECommendation (TaTaRec) for crowdsourcing systems. Our task recommendation framework, TaTaRec, is based on matrix factorization method, to perform factor analysis to learn the worker latent feature, the task latent feature and the task category latent feature with a special constraint of time dimension, while we use an aging scheme in task recommendation to consider mostly fresh and novel worker preferences on tasks. We apply Probabilistic Matrix
Concurrent selection of task and worker
In this section, we present our ActivePMF method (Algorithm 1).
Online-updating approach on model update
In this section, we present our online-updating method for learning the matrix factorization model as presented in Algorithm 1. In the online-updating approach, it has two main parts: (1) It retrains the learning model in batch mode where model update occurs after a number of work done; (2) For each work done related to a worker (or task) having a profile larger than the threshold, it updates the whole feature vector of the worker (or task) and keeps all other entries in the feature matrix
Experimental analysis
In this section, with real-world datasets and synthetic datasets, we intend to address the following eight research questions:
- (1)
How is the TaTaRec compared with the existing state-of-the-art approaches?
- (2)
How does the ActivePMF on TaTaRec compare with PMF with various active learning approaches?
- (3)
How does the ActivePMF on TaTaRec compare with PMF with various active sampling heuristics?
- (4)
How does the aging scheme affect the performance of the ActivePMF on TaTaRec?
- (5)
How does the batch size in the batch
Conclusion and future work
In this paper, we have proposed Time-Aware TAsk RECommendation framework (TaTaRec) in crowdsourcing systems. Our framework performs a factor analysis based on probabilistic matrix factorization (PMF) with which the worker and task latent feature spaces are learned. Our factor analysis model considers both worker task selection preference and worker performance history with a special constraint on the time dimension where the weighting of worker task selection preference gradually decreases over
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgment
This research was in part supported by grants from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. CUHK 14208815 and Project No. UGC/FDS15/E02/20).
References (82)
- et al.
Recommender systems survey
Knowl.-Based Syst.
(2013) - et al.
Combining content-based and collaborative filtering for job recommendation system: A cost-sensitive statistical relational learning approach
Knowl.-Based Syst.
(2017) - et al.
Member contribution-based group recommender system
Decis. Support Syst.
(2016) - et al.
A cross-domain recommender system with consistent information transfer
Decis. Support Syst.
(2017) - et al.
Personalized task recommendation in crowdsourcing information systems — Current state of the art
Decis. Support Syst.
(2014) - et al.
Evolution of the R software ecosystem: Metrics, relationships, and their impact on qualities
J. Syst. Softw.
(2017) - et al.
Pair-wise preference relation based probabilistic matrix factorization for collaborative filtering in recommender system
Knowl.-Based Syst.
(2020) - et al.
Probabilistic matrix factorization with personalized differential privacy
Knowl.-Based Syst.
(2019) - et al.
Real-time recommendation algorithms for crowdsourcing systems
Appl. Comput. Inform.
(2017) - et al.
A deep reinforcement learning based long-term recommender system
Knowl.-Based Syst.
(2021)
A recommender system for component-based applications using machine learning techniques
Knowl.-Based Syst.
A time-aware spatio-textual recommender system
Expert Syst. Appl.
Ticrec: A probabilistic framework to utilize temporal influence correlations for time-aware location recommendations
IEEE Trans. Serv. Comput.
TCARS: time-and community-aware recommendation system
Future Gener. Comput. Syst.
The rise of crowdsourcing
Wired
Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business
A survey of crowdsourcing systems
A survey of human computation systems
The evolution of human computation: Past, present and future
Prog. Hum. Comput. Interact.
A study of human computation applications
Ann. Biostat. Biom. Appl.
Quality management on amazon mechanical turk
Task search in a human computation market
Probabilistic matrix factorization
Online-updating regularized kernel matrix factorization models for large-scale recommender systems
An online-updating algorithm on probabilistic matrix factorization with active learning for task recommendation in crowdsourcing systems
Big Data Anal.
Improving recommender systems by incorporating social contextual information
ACM Trans. Inf. Syst.
Learning to recommend with explicit and implicit social relations
ACM Trans. Intell. Syst. Technol.
A unified point-of-interest recommendation framework in location-based social networks
ACM Trans. Intell. Syst. Technol.
Cited by (22)
A method of user recruitment and adaptation degree improvement via community collaboration in sparse mobile crowdsensing systems
2024, Engineering Applications of Artificial IntelligenceCompetitive gamification in crowdsourcing-based contextual-aware recommender systems
2023, International Journal of Human Computer StudiesA heterogeneous multi-relations-based method for knowledge-intensive service recommendation
2023, Computers and Industrial EngineeringAn optimization approach for worker selection in crowdsourcing systems
2022, Computers and Industrial EngineeringCitation Excerpt :Goel and Faltings (2018) proposed a transitivity mechanism based on golden tasks to incentive workers and measure trust. Yuen et al. (2021) developed a factor analysis model which considers performance history. Zhang and Su (2019) used the combined fuzzy DEMATEL method and TOPSIS method to assess participants in knowledge-intensive crowdsourcing.
TDMatcher: A topic-based approach to task-developer matching with predictive intelligence for recommendation
2021, Applied Soft ComputingCitation Excerpt :Considering the changing of developers’ profiles, Wang et al. [16] designed a skill prediction method based on the learning curve. Yuen et al. [17] built a time-aware task recommendation framework with temporal factors via adding a special constraint on the time dimension. To improve the output quality in crowdsourcing systems, Jung et al. [18] proposed Probabilistic Matrix Factorization for task routing.
- 1
We worked as a team and have been involved in every part of the work.