Skip to main content

Advertisement

Springer Nature Link
Log in

A Dual-Embedding Based Reinforcement Learning Scheme for Task Assignment Problem in Spatial Crowdsourcing

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

With the popularity of mobile devices, spatial crowdsourcing has attracted widespread attention, which collects spatial tasks with location constraints and assigns them to workers who can travel to certain locations to participate in and obtain profits. One of the core issues is task assignment, in which tasks should be assigned to proper workers to maximize the overall utilities. In the paper, we consider a Utility-driven Destination-aware Spatial Task Assignment (UDSTA) problem, where the utility of a worker is modeled as the completed task profit minus the worker’s travel cost, which is more realistic and involves route planning while assigning tasks. We prove that this problem is NP-complete and propose a dual-embedding based deep Q-Network (DE-DQN) to sequentially assign tasks to proper workers. Specifically, we design a utility embedding to reflect the top-k utility tasks for workers and worker-task pairs, and a coverage embedding to represent the potential future utility of an assignment action. The state of DQN consists of the utility embedding, remaining workload, and cumulative utility. Besides, the action of this DQN is formed by concatenating the utility and coverage embedding. We also provide an enhanced version called DE-Rainbow by using Rainbow DQN instead of traditional DQN for further optimization. For the first time, we combine the dual embedding with DQN to achieve a multi-task and multi-worker matching and obtain the route plans of workers. Experiments based on both synthetic and real-world datasets indicate that DE-DQN and DE-Rainbow perform well and show significant advantages over the baseline methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Algorithm 2
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data Availability

The open real-world dataset, Gowalla, could be found at http://snap.stanford.edu/data/loc-Gowalla.html.

Notes

  1. http://snap.stanford.edu/data/loc-Gowalla.html

References

  1. Tong, Y., Zhou, Z., Zeng, Y., Chen, L., Shahabi, C.: Spatial crowdsourcing: a survey. The VLDB Journal (VLDBJ) 29(1), 217–250 (2020)

    Article  MATH  Google Scholar 

  2. Dutta, P., Aoki, P.M., Kumar, N., Mainwaring, A., Myers, C., Willett, W., Woodruff, A.: Common sense: participatory urban sensing using a network of handheld air quality monitors. In: ACM Conference on Embedded Networked Sensor Systems (SenSys), pp. 349–350 (2009)

  3. Rana, R.K., Chou, C.T., Kanhere, S.S., Bulusu, N., Hu, W.: Ear-phone: an end-to-end participatory urban noise mapping system. In: ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN), pp. 105–116 (2010)

  4. Santos, M., Pereira, R.L., Leal, A.B.: Gbus-route geotracer. In: IEEE International Workshop on Vehicular Traffic Management for Smart Cities (VTM), pp. 1–6 (2012)

  5. Zhang, X., Gong, H., Xu, Z., Tang, J., Liu, B.: Jam eyes: a traffic jam awareness and observation system using mobile phones. International Journal of Distributed Sensor Networks (IJDSN) 8(12), 921208 (2012)

    Article  Google Scholar 

  6. Xiong, F., Xu, S., Zheng, D.: An investigation of the uber driver reward system in china-an application of a dynamic pricing model. Technol. Anal. Strategic Manag. 33(1), 44–57 (2021)

    Article  MATH  Google Scholar 

  7. Guo, B., Liu, Y., Wang, L., Li, V.O., Lam, J.C., Yu, Z.: Task allocation in spatial crowdsourcing: current state and future directions. IEEE Internet of Things Journal (IoT-J) 5(3), 1749–1764 (2018)

    Article  MATH  Google Scholar 

  8. Duan, L., Zhan, Y., Hu, H., Gong, Y., Wei, J., Zhang, X., Xu, Y.: Efficiently solving the practical vehicle routing problem: A novel joint learning approach. In: ACM Conference on Knowledge Discovery and Data Mining (SIGKDD), pp. 3054–3063 (2020)

  9. Gao, D., Tong, Y., Ji, Y., Xu, K.: Team-oriented task planning in spatial crowdsourcing. In: Asia-Pacific Web and Web-Age Information Management Joint Conference on Web and Big Data (APWeb-WAIM), pp. 41–56 (2017)

  10. Yin, B., Li, J., Wei, X.: Rational task assignment and path planning based on location and task characteristics in mobile crowdsensing. IEEE Transactions on Computational Social Systems (TCSS) 9(3), 781–793 (2022)

    Article  MATH  Google Scholar 

  11. Ni, W., Cheng, P., Chen, L., Lin, X.: Task allocation in dependency-aware spatial crowdsourcing. In: IEEE International Conference on Data Engineering (ICDE), pp. 985–996 (2020)

  12. Shi, D., Tong, Y., Zhou, Z., Song, B., Lv, W., Yang, Q.: Learning to assign: Towards fair task assignment in large-scale ride hailing. In: ACM SIGKDD Conference on Knowledge Discovery and Data Mining (SIGKDD), pp. 3549–3557 (2021)

  13. Hessel, M., Modayil, J., van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M.G., Silver, D.: Rainbow: Combining improvements in deep reinforcement learning. In: AAAI Conference on Artificial Intelligence (AAAI), pp. 3215–3222 (2018)

  14. Li, M., Cheng, W., Liu, K., He, Y., Li, X., Liao, X.: Sweep coverage with mobile sensors. IEEE Transactions on Mobile Computing (TMC) 10(11), 1534–1545 (2011)

    Article  MATH  Google Scholar 

  15. Bunte, S., Kliewer, N.: An overview on vehicle scheduling models. Public Transport 1(4), 299–317 (2009)

    Article  MATH  Google Scholar 

  16. Gao, G., Wu, J., Xiao, M., Chen, G.: Combinatorial multi-armed bandit based unknown worker recruitment in heterogeneous crowdsensing. In: IEEE International Conference on Computer Communications (INFOCOM), pp. 179–188 (2020)

  17. Zhao, Y., Zheng, K., Cui, Y., Su, H., Zhu, F., Zhou, X.: Predictive task assignment in spatial crowdsourcing: a data-driven approach. In: IEEE International Conference on Data Engineering (ICDE), pp. 13–24 (2020)

  18. Lin, Q., Deng, L., Sun, J., Chen, M.: Optimal demand-aware ride-sharing routing. In: IEEE International Conference on Computer Communications (INFOCOM), pp. 2699–2707 (2018)

  19. Zheng, Y., Wang, J., Li, G., Cheng, R., Feng, J.: QASCA: A quality-aware task assignment system for crowdsourcing applications. In: ACM International Conference on Management of Data (SIGMOD), pp. 1031–1046 (2015)

  20. Cheng, P., Chen, L., Ye, J.: Cooperation-aware task assignment in spatial crowdsourcing. In: IEEE International Conference on Data Engineering (ICDE), pp. 1442–1453 (2019)

  21. Du, Y., Sun, Y.-E., Huang, H., Huang, L., Xu, H., Wu, X.: Quality-aware online task assignment mechanisms using latent topic model. Theoretical Computer Science (TCS) 803, 130–143 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  22. Cheung, M.H., Hou, F., Huang, J., Southwell, R.: Distributed time-sensitive task selection in mobile crowdsensing. IEEE Transactions on Mobile Computing (TMC) 20(6), 2172–2185 (2021)

    Article  MATH  Google Scholar 

  23. Liu, C.H., Zhao, Y., Dai, Z., Yuan, Y., Wang, G., Wu, D., Leung, K.K.: Curiosity-driven energy-efficient worker scheduling in vehicular crowdsourcing: A deep reinforcement learning approach. In: IEEE International Conference on Data Engineering (ICDE), pp. 25–36 (2020)

  24. Zhao, Y., Xia, J., Liu, G., Su, H., Lian, D., Shang, S., Zheng, K.: Preference-aware task assignment in spatial crowdsourcing. In: AAAI Conference on Artificial Intelligence (AAAI), pp. 2629–2636 (2019)

  25. Wang, X., Jia, R., Tian, X., Gan, X.: Dynamic task assignment in crowdsensing with location awareness and location diversity. In: IEEE International Conference on Computer Communications (INFOCOM), pp. 2420–2428 (2018)

  26. Hettiachchi, D., Kostakos, V., Goncalves, J.: A survey on task assignment in crowdsourcing. ACM Comput. Surv. 55(3), 1–35 (2022)

    Article  Google Scholar 

  27. Dai, C., Wang, X., Liu, K., Qi, D., Lin, W., Zhou, P.: Stable task assignment for mobile crowdsensing with budget constraint. IEEE Transactions on Mobile Computing (TMC) 20(12), 3439–3452 (2020)

    Article  MATH  Google Scholar 

  28. Zeng, Y., Tong, Y., Chen, L.: Last-mile delivery made practical: An efficient route planning framework with theoretical guarantees. Proceedings of the VLDB Endowment 13(3), 320–333 (2019)

    Article  Google Scholar 

  29. Tong, Y., Zeng, Y., Ding, B., Wang, L., Chen, L.: Two-sided online micro-task assignment in spatial crowdsourcing. IEEE Trans. Knowl. Data Eng. 33(5), 2295–2309 (2021)

    MATH  Google Scholar 

  30. Tong, Y., Zeng, Y., Zhou, Z., Chen, L., Xu, K.: Unified route planning for shared mobility: An insertion-based framework. ACM Trans. Database Syst. 47(1), 2–1248 (2022)

    Article  MathSciNet  MATH  Google Scholar 

  31. Wang, J., Wang, L., Wang, Y., Zhang, D., Kong, L.: Task allocation in mobile crowd sensing: state-of-the-art and future opportunities. IEEE Internet of Things Journal (IoT-J) 5(5), 3747–3757 (2018)

    Article  MATH  Google Scholar 

  32. Liu, Q., Peng, J., Ihler, A.T.: Variational inference for crowdsourcing. Advances in Neural Information Processing Systems (NeurIPS) 25 (2012)

  33. Wang, H.-n., Liu, N., Zhang, Y.-y., Feng, D.-w., Huang, F., Li, D.-s., Zhang, Y.-m.: Deep reinforcement learning: a survey. Front. Inform. Technol. Electr. Eng. 21(12), 1726–1744 (2020)

  34. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M.A., Fidjeland, A., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)

    Article  Google Scholar 

  35. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems (NeurIPS) 25 (2012)

  36. Shan, C., Mamoulis, N., Cheng, R., Li, G., Li, X., Qian, Y.: An end-to-end deep rl framework for task arrangement in crowdsourcing platforms. In: IEEE International Conference on Data Engineering (ICDE), pp. 49–60 (2020)

  37. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D.: Continuous control with deep reinforcement learning. In: International Conference on Learning Representations (ICLR) (2016)

  38. Sheng, V.S., Zhang, J.: Machine learning with crowdsourcing: a brief summary of the past research and future directions. In: AAAI Conference on Artificial Intelligence (AAAI), vol. 33, pp. 9837–9843 (2019)

  39. Zhao, P., Li, X., Gao, S., Wei, X.: Cooperative task assignment in spatial crowdsourcing via multi-agent deep reinforcement learning. Journal of Systems Architecture: Embedded Software Design (JSA) 128, 102551 (2022)

  40. Wang, Y., Liu, C.H., Piao, C., Yuan, Y., Han, R., Wang, G., Tang, J.: Human-drone collaborative spatial crowdsourcing by memory-augmented and distributed multi-agent deep reinforcement learning. In: IEEE International Conference on Data Engineering (ICDE), pp. 459–471 (2022)

  41. Wang, H., Liu, C.H., Dai, Z., Tang, J., Wang, G.: Energy-efficient 3d vehicular crowdsourcing for disaster response by distributed deep reinforcement learning. In: ACM Conference on Knowledge Discovery & Data Mining (SIGKDD), pp. 3679–3687 (2021)

  42. Sun, Y., Liu, M., Huang, L., Xie, N., Zhao, L., Tan, W.: An embedding-based deterministic policy gradient model for spatial crowdsourcing applications. In: International Conference on Computer Supported Cooperative Work in Design (CSCWD), pp. 1268–1274 (2021)

  43. Shan, C., Mamoulis, N., Cheng, R., Li, G., Li, X., Qian, Y.: An end-to-end deep RL framework for task arrangement in crowdsourcing platforms. In: International Conference on Data Engineering (ICDE), pp. 49–60 (2020)

  44. Ye, G., Zhao, Y., Chen, X., Zheng, K.: Task allocation with geographic partition in spatial crowdsourcing. In: ACM International Conference on Information & Knowledge Management (CIKM), pp. 2404–2413 (2021)

  45. Shen, W., He, X., Zhang, C., Ni, Q., Dou, W., Wang, Y.: Auxiliary-task based deep reinforcement learning for participant selection problem in mobile crowdsourcing. In: ACM International Conference on Information & Knowledge Management (CIKM), pp. 1355–1364 (2020)

  46. Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., Kavukcuoglu, K.: Asynchronous methods for deep reinforcement learning. In: International Conference on Machine Learning (ICML), pp. 1928–1937 (2016)

  47. Schulman, J., Levine, S., Abbeel, P., Jordan, M., Moritz, P.: Trust region policy optimization. In: International Conference on Machine Learning (ICML), pp. 1889–1897 (2015)

  48. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv:1707.06347 (2017)

  49. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing atari with deep reinforcement learning. arXiv:1312.5602 (2013)

  50. Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., Riedmiller, M.: Deterministic policy gradient algorithms. In: International Conference on Machine Learning (ICML), pp. 387–395 (2014)

  51. Fujimoto, S., Hoof, H., Meger, D.: Addressing function approximation error in actor-critic methods. In: International Conference on Machine Learning (ICML), pp. 1587–1596 (2018)

  52. Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: International Conference on Machine Learning (ICML), pp. 1861–1870 (2018)

  53. Racanière, S., Weber, T., Reichert, D., Buesing, L., Guez, A., Jimenez Rezende, D., Puigdomènech Badia, A., Vinyals, O., Heess, N., Li, Y., et al.: Imagination-augmented agents for deep reinforcement learning. Advances in neural information processing systems (NeurIPS) 30 (2017)

  54. Bansal, S., Calandra, R., Chua, K., Levine, S., Tomlin, C.: Mbmf: Model-based priors for model-free reinforcement learning. arXiv:1709.03153 (2017)

  55. Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al.: A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science 362(6419), 1140–1144 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  56. Van Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double q-learning. In: AAAI Conference on Artificial Intelligence (AAAI), pp. 2094–2100 (2016)

  57. Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., Freitas, N.: Dueling network architectures for deep reinforcement learning. In: International Conference on Machine Learning (ICML), pp. 1995–2003 (2016)

  58. Schaul, T., Quan, J., Antonoglou, I., Silver, D.: Prioritized experience replay. In: International Conference on Learning Representations (ICLR) (2016)

  59. Fortunato, M., Azar, M.G., Piot, B., Menick, J., Hessel, M., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., Blundell, C., Legg, S.: Noisy networks for exploration. In: International Conference on Learning Representations (ICLR) (2018)

  60. Bellemare, M.G., Dabney, W., Munos, R.: A distributional perspective on reinforcement learning. In: International Conference on Machine Learning (ICML), vol. 70, pp. 449–458 (2017)

  61. Zheng, L., Cheng, P., Chen, L.: Auction-based order dispatch and pricing in ridesharing. In: IEEE International Conference on Data Engineering (ICDE), pp. 1034–1045 (2019)

  62. Gutin, G., Punnen, A.P. (eds.): The Traveling Salesman Problem and Its Variations, (2007)

  63. Cho, E., Myers, S.A., Leskovec, J.: Friendship and mobility: user movement in location-based social networks. In: ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD), pp. 1082–1090 (2011)

  64. Liu, W., Gao, X.: Leveraging social networks to enhance effective coverage for mobile crowdsensing. In: IEEE International Conference on Web Services (ICWS), pp. 389–393 (2020)

Download references

Acknowledgements

Yucen Gao, Dejun Kong and Xiaofeng Gao are from MoE Key Lab of Artificial Intelligence, Department of Computer Science and Engineering, Shanghai Jiao Tong University. Xiaofeng Gao is the corresponding author. The authors want to give special thanks to Wei Liu for her help and contribution to this paper.

Funding

This work was supported by the National Key R&D Program of China [2024YFF0617700, 2023YFB4502400], the National Natural Science Foundation of China [U23A20309, 62272302, 62172276, 62372296, 62272223, U22A2031, 62422207], the Fundamental Research Funds for the Central Universities [2024300349], the Shanghai Municipal Science and Technology Major Project [2021SHZDZX0102], and the CCF-DiDi GAIA Collaborative Research Funds for Young Scholars [202404].

Author information

Authors and Affiliations

  1. Shanghai Jiao Tong University, Shanghai, 200240, China

    Yucen Gao, Dejun Kong, Xiaofeng Gao & Fan Wu

  2. Nanjing University, Nanjing, 210008, China

    Haipeng Dai, Jiaqi Zheng & Guihai Chen

Authors
  1. Yucen Gao
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Dejun Kong
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Haipeng Dai
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Xiaofeng Gao
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Jiaqi Zheng
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Fan Wu
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Guihai Chen
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

Yucen Gao, Dejun Kong, and Xiaofeng Gao wrote the main manuscript text. Haipeng Dai, Xiaofeng Gao, and Jiaqi Zheng suggested some revisions. All authors reviewed the manuscript.

Corresponding author

Correspondence to Xiaofeng Gao.

Ethics declarations

Ethical Approval

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article belongs to the Topical Collection: APWeb-WAIM 2022

Guest Editors: Calvanese Diego, Toshiyuki Amagasa and Bohan Li

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gao, Y., Kong, D., Dai, H. et al. A Dual-Embedding Based Reinforcement Learning Scheme for Task Assignment Problem in Spatial Crowdsourcing. World Wide Web 28, 13 (2025). https://doi.org/10.1007/s11280-024-01325-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11280-024-01325-9

Keywords