Temporal context-aware task recommendation in crowdsourcing systems

https://doi.org/10.1016/j.knosys.2021.106770Get rights and content

Abstract

In crowdsourcing systems, tasks are distributed to networked people for completion. To ensure the output quality, current crowdsourcing systems highly rely on redundancy of answers provided by multiple workers, however massive redundancy is very expensive. Task recommendation can help requesters to receive good quality output quicker as well as help workers to find their right tasks faster. In our previous works, we proposed a task recommendation framework which performs a factor analysis based on probabilistic matrix factorization (PMF) with which the worker and task latent feature spaces are learned. Our framework adopts active learning on our factor analysis model to minimize the number of task assignments to achieve a target output quality. Moreover, our framework adopts an online-updating approach on model update process to greatly improve the system performance in terms of the running time of model update and the prediction accuracy. However, all previous works on task recommendation in crowdsourcing systems do not consider the temporal change of workers’ preference on tasks, thus cannot make recommendations depending on fresh and novel workers’ preference on tasks. In this paper, we propose a time-aware task recommendation framework in crowdsourcing systems, called Time-Aware TAsk RECommendation (TaTaRec). Our factor analysis model considers both worker task selection preference and worker performance history with a special constraint on the time dimension where the weighting of worker task selection preference gradually decreases over time. Complexity analysis shows that our model is efficient and is scalable to large datasets. We carry out comprehensive experiments on our framework by using both real-world datasets and synthetic datasets to evaluate the performance of our framework and the effects of various parameters on the behaviors of our framework. To the best of our knowledge, we are the first one to propose a task recommendation framework that considers the time aspect of workers’ preference on tasks.

Introduction

Crowdsourcing is an idea of outsourcing a task to a large group of networked people in the form of an open call to reduce the production cost [1], [2]. In recent years, crowdsourcing systems attract much attention at present [3], [4], [5], [6]. Some examples of crowdsourcing systems are Amazon Mechanical Turk (or MTurk) [7], CrowdFlower [8], Taskcn [9] and TopCoder [10]. To achieve quality assurance in crowdsourcing systems, a requester has to verify the quality of every answer submitted by workers, and it is very time-consuming. Alternatively, requesters highly rely on redundancy of answers provided by multiple workers with varying expertise, but massive redundancy is very expensive and time-consuming. “If we ask 10 workers to complete the same task, then the cost of crowdsourcing solutions tends to be comparable to the cost of in-house solutions” [11]. Therefore, it is important to investigate how to enable task requesters to verify correct answers on crowdsourcing platforms more easily and effectively. Chilton et al. [12] showed that task workers look mostly at the first page of the most recently posted tasks and the first two pages of the tasks with the most available instances. The worker performance history including workers’ task searching behavior makes it possible to mine workers’ preference on tasks and to provide an indication of worker quality on tasks. Therefore, based on worker performance histories, task recommendation can help to provide a list of preferred tasks to workers in crowdsourcing systems to guarantee the quality of work done. However, workers’ preference on tasks might change from time to time, so it is necessary to consider the temporal change of workers’ preference on tasks when making task recommendations.

In addition, it is important to help workers to find their right tasks as quick as possible and minimize the number of task assignments to achieve a target output quality [13]. Active learning can be applied on task recommendation to help requesters to receive good quality output quicker with lower cost, thus guaranteeing quality in a shorter period of time.

Probabilistic Matrix Factorization (PMF) [14] is the state-of-the-art approach for recommendation systems. A factorization model has to be trained and learned before the model can be applied for prediction. In real-world applications, the performance of a factorization model is highly affected by how the model is updated, and thus dynamic updating a model is very important [15].

In our previous works [16], [17], we proposed a task recommendation framework which performs a factor analysis based on probabilistic matrix factorization (PMF) with which the worker and task latent feature spaces are learned. Our framework adopts an active learning approach to achieve a target output quality with the minimum task assignments. Moreover, our framework uses an online-updating method on model update process to greatly improve the system performance in terms of the running time of model update and the prediction accuracy.

Workers’ preference might change over time, but all the research works in the literature in task recommendation do not consider the temporal change of workers’ preference on tasks. As a result, they cannot make recommendations depending on fresh and novel workers’ preference on tasks.

Our contributions are:

  • to propose a method for task recommendation by performing unified matrix factorization on worker-task preference matrix, worker-category preference matrix and task-category grouping matrix with which the worker and task latent feature spaces are learned with a special constraint on the time dimension,

  • to propose an aging scheme for task recommendation by depending on mostly fresh and novel worker preferences in order to reflect the change of workers’ opinions in real world applications,

  • to propose an active learning method for task recommendation by selecting the most informative task and the most skilful worker in order to guarantee the output quality of the factor analysis model,

  • to propose an online-updating method for task recommendation in order to reduce the cost of model update,

  • to propose a generic online-updating method when learning a factor analysis model,

  • to show that our model is efficient and is scalable to large datasets by means of complexity analysis,

  • to demonstrate the performance of our task recommendation framework by using real-world datasets,

  • to study the effects of various parameters on our task recommendation framework by using synthetic datasets.

To the best of our knowledge, our work is the first research work considering temporal factors in task recommendation. The rest of this paper is organized as follows. Section 2 presents the related works. Section 3 presents our proposed time-aware task recommendation framework in crowdsourcing systems. Section 4 presents our proposed active learning method. Section 5 presents our proposed online-updating method. Section 6 describes our experiments. Section 7 concludes with future directions.

Section snippets

Related work

Recommendation systems can model users’ interest and perform interest-based user recommendation. With the exponential growth of information on the Web, social recommendation has attracted much attention [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36]. Social recommendation uses computational techniques (such as machine learning and data mining) on social behavior data collected from blogs and tags, etc. from social media (such as

Time-aware task recommendation framework

We propose a framework for Time-Aware TAsk RECommendation (TaTaRec) for crowdsourcing systems. Our task recommendation framework, TaTaRec, is based on matrix factorization method, to perform factor analysis to learn the worker latent feature, the task latent feature and the task category latent feature with a special constraint of time dimension, while we use an aging scheme in task recommendation to consider mostly fresh and novel worker preferences on tasks. We apply Probabilistic Matrix

Concurrent selection of task and worker

In this section, we present our ActivePMF method (Algorithm 1).

Online-updating approach on model update

In this section, we present our online-updating method for learning the matrix factorization model as presented in Algorithm 1. In the online-updating approach, it has two main parts: (1) It retrains the learning model in batch mode where model update occurs after a number of work done; (2) For each work done related to a worker (or task) having a profile larger than the threshold, it updates the whole feature vector of the worker (or task) and keeps all other entries in the feature matrix

Experimental analysis

In this section, with real-world datasets and synthetic datasets, we intend to address the following eight research questions:

  • (1)

    How is the TaTaRec compared with the existing state-of-the-art approaches?

  • (2)

    How does the ActivePMF on TaTaRec compare with PMF with various active learning approaches?

  • (3)

    How does the ActivePMF on TaTaRec compare with PMF with various active sampling heuristics?

  • (4)

    How does the aging scheme affect the performance of the ActivePMF on TaTaRec?

  • (5)

    How does the batch size in the batch

Conclusion and future work

In this paper, we have proposed Time-Aware TAsk RECommendation framework (TaTaRec) in crowdsourcing systems. Our framework performs a factor analysis based on probabilistic matrix factorization (PMF) with which the worker and task latent feature spaces are learned. Our factor analysis model considers both worker task selection preference and worker performance history with a special constraint on the time dimension where the weighting of worker task selection preference gradually decreases over

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgment

This research was in part supported by grants from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. CUHK 14208815 and Project No. UGC/FDS15/E02/20).

References (82)

  • Fernández-GarcíaA.J. et al.

    A recommender system for component-based applications using machine learning techniques

    Knowl.-Based Syst.

    (2019)
  • KefalasP. et al.

    A time-aware spatio-textual recommender system

    Expert Syst. Appl.

    (2017)
  • ZhangJ.-D. et al.

    Ticrec: A probabilistic framework to utilize temporal influence correlations for time-aware location recommendations

    IEEE Trans. Serv. Comput.

    (2015)
  • RezaeimehrF. et al.

    TCARS: time-and community-aware recommendation system

    Future Gener. Comput. Syst.

    (2018)
  • HoweJ.

    The rise of crowdsourcing

    Wired

    (2006)
  • HoweJ.

    Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business

    (2008)
  • YuenM.-C. et al.

    A survey of crowdsourcing systems

  • YuenM.-C. et al.

    A survey of human computation systems

  • YuenM.-C.

    The evolution of human computation: Past, present and future

    Prog. Hum. Comput. Interact.

    (2018)
  • YuenM.-C.

    A study of human computation applications

    Ann. Biostat. Biom. Appl.

    (2018)
  • Amazon Mechanical Turk,...
  • CrowdFlower,...
  • Taskcn,...
  • TopCoder,...
  • IpeirotisP.G. et al.

    Quality management on amazon mechanical turk

  • ChiltonL.B. et al.

    Task search in a human computation market

  • D.R. Karger, S. Oh, D. Shah, Iterative Learning for Reliable Crowdsourcing Systems, in: Advances in Neural Information...
  • SalakhutdinovR. et al.

    Probabilistic matrix factorization

  • RendleS. et al.

    Online-updating regularized kernel matrix factorization models for large-scale recommender systems

  • YuenM.-C. et al.

    An online-updating algorithm on probabilistic matrix factorization with active learning for task recommendation in crowdsourcing systems

    Big Data Anal.

    (2016)
  • M.-C. Yuen, I. King, K.-S. Leung, An Online-Updating Approach on Task Recommendation in Crowdsourcing Systems, in:...
  • I. King, M.R. Lyu, H. Ma, Introduction to social recommendation, in: Proceedings of the 19th International Conference...
  • H. Ma, I. King, M.R. Lyu, Learning to recommend with social trust ensemble, in: Proceedings of the 32nd Annual...
  • H. Ma, M.R. Lyu, I. King, Learning to recommend with trust and distrust relationships, in: RecSys ’09: Proceedings of...
  • X. Xin, I. King, H. Deng, M.R. Lyu, A social recommendation framework based on multi-scale continuous conditional...
  • T.C. Zhou, H. Ma, M.R. Lyu, I. King, UserRec: A User Recommendation Framework in Social Tagging Systems, in: AAAI ’10:...
  • H. Ma, D. Zhou, C. Liu, M.R. Lyu, I. King, Recommender systems with social regularization, in: WSDM ’11: Proceedings of...
  • MaH. et al.

    Improving recommender systems by incorporating social contextual information

    ACM Trans. Inf. Syst.

    (2011)
  • MaH. et al.

    Learning to recommend with explicit and implicit social relations

    ACM Trans. Intell. Syst. Technol.

    (2011)
  • H. Ma, C. Liu, I. King, M.R. Lyu, Probabilistic factor models for web site recommendation, in: SIGIR ’11: Proceeding of...
  • ChengC. et al.

    A unified point-of-interest recommendation framework in location-based social networks

    ACM Trans. Intell. Syst. Technol.

    (2016)
  • Cited by (22)

    • An optimization approach for worker selection in crowdsourcing systems

      2022, Computers and Industrial Engineering
      Citation Excerpt :

      Goel and Faltings (2018) proposed a transitivity mechanism based on golden tasks to incentive workers and measure trust. Yuen et al. (2021) developed a factor analysis model which considers performance history. Zhang and Su (2019) used the combined fuzzy DEMATEL method and TOPSIS method to assess participants in knowledge-intensive crowdsourcing.

    • TDMatcher: A topic-based approach to task-developer matching with predictive intelligence for recommendation

      2021, Applied Soft Computing
      Citation Excerpt :

      Considering the changing of developers’ profiles, Wang et al. [16] designed a skill prediction method based on the learning curve. Yuen et al. [17] built a time-aware task recommendation framework with temporal factors via adding a special constraint on the time dimension. To improve the output quality in crowdsourcing systems, Jung et al. [18] proposed Probabilistic Matrix Factorization for task routing.

    View all citing articles on Scopus
    1

    We worked as a team and have been involved in every part of the work.

    View full text