Abstract
Mobile sensing has become a promising paradigm for monitoring the environmental state. When equipped with sensors, a group of unmanned vehicles can autonomously move around for distributed sensing. To maximize the sensing coverage, a critical challenge is to coordinate the decentralized vehicles for cooperation. In this work, we propose a novel algorithm Comm-Q, in which the vehicles can learn to communicate for cooperation via multi-agent reinforcement learning. At each step, the vehicles can broadcast a message to others, and condition on received aggregated message to update their sensing policies. The message is also learned via reinforcement learning. In addition, we decompose and reshape the reward function for more efficient policy training. Experimental results show that our algorithm is scalable and can converge very fast during training phase. It also outperforms other baselines significantly during execution. The results validate that communication message plays an important role to coordinate the behaviors of different vehicles.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Carnelli, P.E., Yeh, J., Sooriyabandara, M., Khan, A.: Parkus: a novel vehicle parking detection system. In: Twenty-Ninth IAAI Conference (2017)
Das, A., et al.: Tarmac: Targeted multi-agent communication. In: International Conference on Machine Learning, pp. 1538–1546. PMLR (2019)
Foerster, J., Assael, I.A., de Freitas, N., Whiteson, S.: Learning to communicate with deep multi agent reinforcement learning. Adv. Neural. Inf. Process. Syst. 29, 2137–2145 (2016)
Foerster, J.N., Farquhar, G., Afouras, T., Nardelli, N., Whiteson, S.: Counterfactual multi-agent policy gradients. In: AAAI, pp. 2974–2982 (2018)
Guo, B., et al.: Mobile crowd sensing and computing: The review of an emerging human-powered sensing paradigm. ACM Comput. Surv. (CSUR) 48(1), 1–31 (2015)
Jaques, N., et al.: Intrinsic social motivation via causal influence in multi-agent RL (2018)
Karaliopoulos, M., Telelis, O., Koutsopoulos, I.: User recruitment for mobile crowdsensing over opportunistic networks. In: 2015 IEEE Conference on Computer Communications (INFOCOM), pp. 2254–2262. IEEE (2015)
Lane, N.D., Miluzzo, E., Lu, H., Peebles, D., Choudhury, T., Campbell, A.T.: A survey of mobile phone sensing. IEEE Commun. Mag. 48(9), 140–150 (2010)
Liu, C.H., Ma, X., Gao, X., Tang, J.: Distributed energy-efficient multi-UAV navigation for long-term communication coverage by deep reinforcement learning. IEEE Trans. Mob. Comput. 19(6), 1274–1285 (2019)
Lowe, R., Foerster, J., Boureau, Y.L., Pineau, J., Dauphin, Y.: On the pitfalls of measuring emergent communication. arXiv preprint arXiv:1903.05168 (2019)
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, O.P., Mordatch, I.: Multi-agent actor-critic for mixed cooperative-competitive environments. Adv. Neural. Inf. Process. Syst. 30, 6379–6390 (2017)
Mnih, V., et al.: Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)
Rashid, T., Samvelyan, M., Schroeder, C., Farquhar, G., Foerster, J., Whiteson, S.: Qmix: monotonic value function factorisation for deep multi-agent reinforcement learning. In: International Conference on Machine Learning, pp. 4292–4301 (2018)
Silver, D., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016)
Sukhbaatar, S., Szlam, A., Fergus, R.: Learning multiagent communication with backpropagation. arXiv preprint arXiv:1605.07736 (2016)
Sunehag, P., et al.: Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296 (2017)
Sutton, R.S., Barto, A.G., et al.: Introduction to Reinforcement Learning, vol. 135. MIT Press, Cambridge (1998)
Tan, M.: Multi-agent reinforcement learning: independent vs. cooperative agents. In: ICML 1993 Proceedings of the Tenth International Conference on International Conference on Machine Learning, pp. 487–494 (1997)
Zhou, Z., et al.: When mobile crowd sensing meets UAV: energy-efficient task assignment and route planning. IEEE Trans. Commun. 66(11), 5526–5538 (2018)
Acknowledgments
This research was funded by Natural Science Foundation of Jiangsu Province (No. BK20200752); The NUPTSF (No. NY220080).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, B., Liu, J., Xiao, F. (2021). Learning to Communicate for Mobile Sensing with Multi-agent Reinforcement Learning. In: Liu, Z., Wu, F., Das, S.K. (eds) Wireless Algorithms, Systems, and Applications. WASA 2021. Lecture Notes in Computer Science(), vol 12938. Springer, Cham. https://doi.org/10.1007/978-3-030-86130-8_48
Download citation
DOI: https://doi.org/10.1007/978-3-030-86130-8_48
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86129-2
Online ISBN: 978-3-030-86130-8
eBook Packages: Computer ScienceComputer Science (R0)