Skip to main content

Advertisement

Springer Nature Link
Log in

Online budget-feasible mobile crowdsensing with constrained reinforcement learning

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Mobile crowdsensing, which leverages distributed mobile workers to collect geospatial data, has facilitated a wide range of applications, such as fire detection, traffic monitoring, and air quality sensing. However, due to the dynamics of the environment, it remains a critical challenge to incentivize high-quality workers in an online environment with limited budget. In this paper, we study the mobile crowdsensing problem with budget constraint in both offline and online setting. We first investigate a simplified offline setting where the workers’ valuation functions are known but probabilistic. The problem can be proved NP-hard under this setting. To deal with the hardness, we propose a smooth-greedy algorithm to allocate the budget, which has constant approximation ratio. In the online setting where the workers’ valuations are unknown and dynamic, we model the problem as a Markov decision process: The aim is to maximize long-term cumulative reward by allocating limited budget iteratively. Motivated by the smooth-greedy idea, we decompose the reward function for each individual worker to reduce the action space, and further develop a constrained reinforcement learning method. The algorithm uses primal-dual method to find the optimal mechanism while ensuring the budget constraint. Extensive experiments are implemented to evaluate the performance of our algorithms. The results validate that our method can achieve the best performance in almost all cases. The performance is comparable to the approximate solution even when the valuation functions are known.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Fig. 3
Algorithm 2
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

The dataset can be accessed in the following link: https://www.dropbox.com/s/42cl68ns2fud5yk/GOOGLETraffic.zip?dl=0.

Code availability

The codes are not publicly available.

Notes

  1. The input of \(U(\cdot )\) can be either set or vector. We slightly abuse notations for simplicity.

  2. Dataset collected from google map: https://www.dropbox.com/s/42cl68ns2fud5yk/GOOGLETraffic.zip?dl=0.

References

  1. Achiam J, Held D, Tamar A et al (2017) Constrained policy optimization. In: International conference on machine learning, PMLR, pp 22–31

  2. An N, Wang R, Luan Z et al (2015) Adaptive assignment for quality-aware mobile sensing network with strategic users. In: 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security, and 2015 IEEE 12th International Conference on Embedded Software and Systems, IEEE, pp 541–546

  3. Balcan MF, Constantin F, Iwata S et al (2012) Learning valuation functions. In: Conference on Learning Theory, JMLR Workshop and Conference Proceedings, pp 4–1

  4. Cai Z, Duan Z, Li W (2020) Exploiting multi-dimensional task diversity in distributed auctions for mobile crowdsensing. IEEE Trans Mob Comput 20(8):2576–2591

    Article  Google Scholar 

  5. Feng C, Wang W, Tian Y et al (2017) Estimate air quality based on mobile crowd sensing and big data. In: 2017 IEEE 18th International Symposium on a world of Wireless, Mobile and Multimedia Networks (WoWMoM), IEEE, pp 1–9

  6. Gai Y, Krishnamachari B, Jain R (2012) Combinatorial network optimization with unknown variables: multi-armed bandits with linear rewards and individual observations. IEEE/ACM Trans Netw 20(5):1466–1478

    Article  Google Scholar 

  7. Gan X, Li Y, Wang W et al (2017) Social crowdsourcing to friends: an incentive mechanism for multi-resource sharing. IEEE J Sel Areas Commun 35(3):795–808

    Article  Google Scholar 

  8. Gao G, Huang H, Xiao M et al (2021) Budgeted unknown worker recruitment for heterogeneous crowdsensing using cmab. IEEE Trans Mob Comput 21(11):3895–3911

    Google Scholar 

  9. Gao H, Xu H, Li L et al (2022) Mean-field-game-based dynamic task pricing in mobile crowdsensing. IEEE Internet Things J 9(18):18098–18112

    Article  Google Scholar 

  10. Gao H, Xu H, Zhou C et al (2022) Dynamic task pricing in mobile crowdsensing: an age-of-information-based queueing game scheme. IEEE Internet Things J 9(21):21278–21291

    Article  Google Scholar 

  11. Gao X, Chen S, Chen G (2020) Mab-based reinforced worker selection framework for budgeted spatial crowdsensing. IEEE Trans Knowl Data Eng 34(3):1303–1316

    Article  Google Scholar 

  12. Haarnoja T, Zhou A, Hartikainen K et al (2018) Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905

  13. Han K, Huang H, Luo J (2018) Quality-aware pricing for mobile crowdsensing. IEEE/ACM Trans Netw 26(4):1728–1741

    Article  Google Scholar 

  14. Ul Hassan U, Curry E (2016) Efficient task assignment for spatial crowdsourcing: a combinatorial fractional optimization approach with semi-bandit learning. Expert Syst Appl 58:36–56

    Article  Google Scholar 

  15. Huo J, Wang L, Wen X et al (2023) Cost-efficient vehicular crowdsensing based on implicit relation aware graph attention networks. IEEE Trans Ind Inform

  16. Ioannidis S, Chaintreau A, Massoulié L (2009) Optimal and scalable distribution of content updates over a mobile social network. In: IEEE INFOCOM 2009, IEEE, pp 1422–1430

  17. Ji G, Yao Z, Zhang B et al (2020) A reverse auction-based incentive mechanism for mobile crowdsensing. IEEE Internet Things J 7(9):8238–8248

    Article  Google Scholar 

  18. Jin H, Su L, Nahrstedt K (2017) Centurion: Incentivizing multi-requester mobile crowd sensing. In: IEEE INFOCOM 2017-IEEE Conference on Computer Communications, IEEE, pp 1–9

  19. Karaliopoulos M, Telelis O, Koutsopoulos I (2015) User recruitment for mobile crowdsensing over opportunistic networks. In: 2015 IEEE Conference on Computer Communications (INFOCOM), IEEE, pp 2254–2262

  20. Liu CH, Dai Z, Zhao Y et al (2019) Distributed and energy-efficient mobile crowdsensing with charging stations by deep reinforcement learning. IEEE Trans Mob Comput 20(1):130–146

    Article  Google Scholar 

  21. Marshall A (2009) Principles of economics: unabridged. 8th edn. Cosimo, Inc

  22. Mnih V, Kavukcuoglu K, Silver D et al (2013) Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602

  23. Nadembega A, Hafid AS, Brisebois R (2016) Mobility prediction model-based service migration procedure for follow me cloud to support QOS and QOE. In: 2016 IEEE International Conference on Communications (ICC), pp 1–6

  24. Paternain S, Chamon L, Calvo-Fullana M et al (2019) Constrained reinforcement learning has zero duality gap. Adv Neural Inf Process Syst 32

  25. Peng XB, Abbeel P, Levine S et al (2018) Deepmimic: example-guided deep reinforcement learning of physics-based character skills. ACM Trans Graphics (TOG) 37(4):1–14

    Google Scholar 

  26. Schulman J, Wolski F, Dhariwal P et al (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

  27. Simon B, Ortiz A, Saad W et al (2024) Decentralized online learning in task assignment games for mobile crowdsensing. IEEE Trans Commun

  28. Sun J, Wu D (2022) Online truthful incentives for heterogeneous and \( k \)-submodular crowdsensing. IEEE Syst J 17(2):3083–3092

    Article  MathSciNet  Google Scholar 

  29. Sun J, Wu D (2023) Contextual features based budget-limited online pricing for heterogeneous sensing tasks. IEEE Internet Things J

  30. Sun J, Wu D (2023) Ellipsoid-pricing-based context-feature mechanisms for noisy sensing tasks. IEEE Internet Things J 10(22):19972–19980

    Article  Google Scholar 

  31. Sutton RS, McAllester D, Singh S et al (1999) Policy gradient methods for reinforcement learning with function approximation. Adv Neural Inf Process Syst 12

  32. Taleb T, Ksentini A (2013) An analytical model for follow me cloud. In: 2013 IEEE Global Communications Conference (GLOBECOM), pp 1291–1296

  33. Truong NB, Lee GM, Um TW et al (2019) Trust evaluation mechanism for user recruitment in mobile crowd-sensing in the internet of things. IEEE Trans Inf Forensics Secur 14(10):2705–2719

    Article  Google Scholar 

  34. Wan J, Liu J, Shao Z et al (2016) Mobile crowd sensing for traffic prediction in internet of vehicles. Sensors 16(1):88

    Article  Google Scholar 

  35. Wang H, Liu CH, Yang H et al (2023) Ensuring threshold AOI for UAV-assisted mobile crowdsensing by multi-agent deep reinforcement learning with transformer. IEEE/ACM Trans Netw

  36. Wang N, Wu J (2018) Cost-efficient heterogeneous worker recruitment under coverage requirement in spatial crowdsourcing. IEEE Trans Big Data 7(2):407–420

    Article  Google Scholar 

  37. Wu Y, Li F, Ma L et al (2019) A context-aware multiarmed bandit incentive mechanism for mobile crowd sensing systems. IEEE Internet Things J 6(5):7648–7658

    Article  Google Scholar 

  38. Xu C, Song W (2023) Decentralized task assignment for mobile crowdsensing with multi-agent deep reinforcement learning. IEEE Internet Things J 10(18):16564–16578

    Article  Google Scholar 

  39. Xu C, Song W (2023) Intelligent task allocation for mobile crowdsensing with graph attention network and deep reinforcement learning. IEEE Trans Netw Sci Eng 10(2):1032–1048

    Article  Google Scholar 

  40. Xu Z, Mei L, Choo KKR et al (2018) Mobile crowd sensing of human-like intelligence using social sensors: a survey. Neurocomputing 279:3–10

    Article  Google Scholar 

  41. Yang D, Xue G, Fang X et al (2012) Crowdsourcing to smartphones: incentive mechanism design for mobile phone sensing. In: Proceedings of the 18th Annual International Conference on Mobile Computing and Networking, pp 173–184

  42. Yang G, Guo D, Wang B et al (2023) Participant-quantity-aware online task allocation in mobile crowd sensing. IEEE Internet Things J

  43. Yang S, Wu F, Tang S et al (2016) Selecting most informative contributors with unknown costs for budgeted crowdsensing. In: 2016 IEEE/ACM 24th International Symposium on Quality of Service (IWQoS), IEEE, pp 1–6

  44. Zhao H, Xiao M, Wu J et al (2020) Differentially private unknown worker recruitment for mobile crowdsensing using multi-armed bandits. IEEE Trans Mob Comput 20(9):2779–2794

    Article  Google Scholar 

  45. Zhao Y, Liu CH (2020) Social-aware incentive mechanism for vehicular crowdsensing by deep reinforcement learning. IEEE Trans Intell Transp Syst 22(4):2314–2325

    Article  Google Scholar 

  46. Zhou D, Tomlin C (2018) Budget-constrained multi-armed bandits with multiple plays. In: Proceedings of the AAAI Conference on Artificial Intelligence

  47. Zhu X, Luo Y, Liu A et al (2020) A deep learning-based mobile crowdsensing scheme by predicting vehicle mobility. IEEE Trans Intell Transp Syst 22(7):4648–4659

    Article  Google Scholar 

Download references

Funding

This research was supported by National Natural Science Foundation of China under Grant No. 62202238, and China Postdoctoral Science Foundation No. 2024M751506.

Author information

Authors and Affiliations

  1. School of Computer, Nanjing University of Posts and Telecommunications, Nanjing, China

    Bolei Zhang & Lifa Wu

Authors
  1. Bolei Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Lifa Wu
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

B.Z. wrote the code used in this project, with the discussion of L.W. Both B.Z. and L.W. contributed equally to the manuscript. B.Z is the corresponding author.

Corresponding author

Correspondence to Bolei Zhang.

Ethics declarations

Conflict of interest

The authors have no conflict of interest as defined by Springer, or other interests that might be perceived to influence the results and/or discussion reported in this paper.

Ethical approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, B., Wu, L. Online budget-feasible mobile crowdsensing with constrained reinforcement learning. J Supercomput 81, 286 (2025). https://doi.org/10.1007/s11227-024-06767-6

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11227-024-06767-6

Keywords