Abstract
This paper proposes a reinforcement learning-based collaborative multi-agent actor and critic scheme (RL-CMAS) under edge computing architecture for emergency vehicle preemption. The RL-CMAS deployed a parallel training process at the cloud side for building knowledge and well accelerating learning. Priority of message and model of message offloading strategy have been developed. The simulation results show that the proposed RL-CMAS is efficient in detecting even complex data. Finally, a comparison was made with other benchmark methods, namely, Regular scheduling algorithm, Alameddine’s DTOS algorithm, and independent multi-agent actor-critic. The result showed the proposed method outperforming the other three bench marking methods. The proposed RL-CMAS provides reduction in message processing delay, total delay, and an increase of message delivery success ratio of 14.22%, 18.21%, and 8.86% respectively.

















Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Ian, E., Blanchard, C.J., Doig, B.E., Hagel, A.R., Anton, D.A., Zygun, J.B., Kortbeek, D., Gregory Powell, T.S., Williamson, G.H., Fick, Grant, D., Innes: Emergency medical services response time and mortality in an urban setting. Prehospital Emerg. Care 16(1), 142–151 (2012). https://doi.org/10.3109/10903127.2011.614046
He, Q., Head, K.L., Ding, J.: Multi-modal traffic signal control with priority, signal actuation and coordination. Transp. Res. Part C. 46, 65–82 (2014)
Guler, S.I., Menendez, M.: Analytical formulation and empirical evaluation of pre-signals for bus priority. Transp. Res. Part. B 64, 41–53 (2014)
Bell, M.G., Fonzone, A., Polyzoni, C.: Depot location in degradable transport networks. Transp. Res Part B 66, 148–161 (2014)
Salazar-González, J.J., Santos-Hernández, B.: The split-demand one-commodity pickup-and-delivery travelling salesman problem. Transp. Res. Part. B 75, 58–73 (2015)
Liu, K., Li, Q., Zhang, Z.H.: Distributionally robust optimization of an emergency medical service station location and sizing problem with joint chance constraints. Transp. Res. Part. B 119, 79–101 (2019)
He, Q., Irnich, S., Song, Y.: Branch-and-cut-and-price for the vehicle routing problem with time windows and convex node costs. Transp. Sci 53(5), 1409–1426 (2019)
Ansari, S., McLay, L.A., Mayorga, M.E.: A maximum expected covering problem for district design. Transp. Sci 51(1), 376–390 (2015)
Huang, Y.S., Weng, Y.S., Zhou, M.: Design of traffic safety control systems for emergency vehicle preemption using timed petri nets. IEEE Trans. Intell. Transp. Syst 16(4), 2113–2120 (2015)
Hannoun, G.J., Murray-Tuite, P., Heaslip, K., Chantem, T.: Facilitating emergency response vehicles’ movement through a road segment in a connected vehicle environment. IEEE Trans. Intell. Transp. Syst 20(9), 3546–3557 (2018)
Chen, D., Ahn, S., Laval, J., Zheng, Z.: On the periodicity of traffic oscillations and capacity drop: the role of driver characteristics. Transp. Res. Part. B 59, 117–136 (2014)
Rosayyan, P., Subramaniam, S., Ganesan, S.I.: Decentralized emergency service vehicle pre-emption system using RF communication and GNSS-based geo- fencing. IEEE Transactions on Intelligent Transportation Systems, pp. 1–10. https://doi.org/10.1109/TITS.2020.3007671
Mukherjee, M., Shu, L., Wang, D.: Survey of fog omputing: fundamental, network applications, and research challenges. IEEE Commun. Surv. Tutor 20(3), 1826–1857 (2018)
Wu, Q., Shen, J., Yong, B., Wu, J., Li, F., Wang, J., Zhou, Q.: Smart fog based workflow for traffic control networks. Future Gener. Comput. Syst 97, 825–835 (2019)
Liu, J., Li, L., Zhang, F., Dai, Y., Zhang, X., Meng, J., Shen: Secure intelligent traffic light control using fog computing. Future Gener. Comput. Syst 78, 817–824 (2017). https://doi.org/10.1016/j.future.2017.02.017
Jinjian, L., Jacques, B., Arnaud, D., Guillaume, L.: Multi-models machine learning methods for traffic flow estimation from floating car data. Transp. Res. Part C: Emerg Technol. 132 (2021). https://doi.org/10.1016/j.trc.2021.103389
Posor, J.E., Belzner, L., Knapp, A.: Joint action learning for multi-agent cooperation using recurrent reinforcement learning. Digitale Welt. 479–84 (2020). https://doi.org/10.1007/s42354-019-0239-y
Sunehag, P., et al.: Value-decomposition networks for cooperative multi-agent learning, 2017, arXiv preprint arXiv:1706.05296, https://arxiv.org/abs/1706.05296
Saleem, M.A., Mahmood, K., Kumari, S.: Comments on AKM-IoV: Authenticated key management protocol in fog computing-based internet of vehicles deployment. IEEE Internet Things J 7(5), 4671–4675 (2020). https://doi.org/10.1109/JIOT.2019.2923611
Sodhro, A.H., Pirbhulal, S., de Albuquerque, V.H.C.: Artificial intelligence driven mechanism for edge computing-based industrial applications. IEEE Trans. Ind. Inform 15(7), 4235–4243 (2019). https://doi.org/10.1109/TII.2019.2902878
Shi, W., Cao, J., Zhang, Q., Li, Y., Xu, L.: Edge computing: vision and challenges. IEEE Internet Things J 3(5), 637–646 (2016)
Zhang, F., Li, J., Li, Z.: A TD3-based multi-agent deep reinforcement learning method in mixed cooperation- competition environment. Neurocomputing 411, 206–215 (2020)
Cao, W., Xu, S., An emergency warning message dissemination policy of vehicular ad hoc network: 9th International Symposium on Computational Intelligence and Design (ISCID), 2016, pp. 227–230 (2016). https://doi.org/10.1109/ISCID.2016.2061
Hollinghurst, J., Ganesh, A., Baugé, T.: Latency reduction in communication networks using redundant messages. Paper presented at: Proceedings of the 2017 29th International Teletraffic Congress (ITC 29), pp. 241–249. (2017)
Kausar, S., Habib, M., Shabir, M.Y., et al.: Secure and efficient data transfer using spreading and assimilation in MANET. Softw. Pract. Exp 1(50), 35–50 (2020)
Wu, X., Chen, H., Chen, C., Zhong, M., Xie, S., Guo, Y., Fujita, H.: The autonomous navigation and obstacle avoidance for USVs with ANOA deep reinforcement learning method. Knowl. -Based Syst 196, 1–12 (2020)
Yang, S., Yang, B., Wong, H., Kang, Z.: Cooperative traffic signal control using multi-step return and off-policy asynchronous advantage actor-critic graph algorithm. Knowl. -Based Syst 183, 1–19 (2019)
Rasheed, F., Yau, K.A., Low, Y.: Deep reinforcement learning for traffic signal control under disturbances: a case study on sunway city, Malaysia, Future Gener. Comput. Syst 109, 431–445 (2020)
Wang, H., Li, J., Chen, Q.Y., Ni, D.: Logistic modeling of equilibrium speed–density relationship. Transp. Res. Part A: Policy Pract 45, 554–566 (2011)
Wei, H., Chen, C., Zheng, G., Wu, K., Gayah, V., Xu, K., Li, Z., Presslight: Learning max pressure control to coordinate traffic signals in arterial network. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1290–1298 (2019)
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Nature 323, 533–536 (1986). https://doi.org/10.1038/323533a0
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, O.P., Mordatch, I.: Multi-agent actor-critic for mixed cooperative- competitive environments. In: Advances in Neural Information Processing Systems, pp. 6379–6390 (2017)
Fedesoriano.: Traffic Prediction Dataset, Version 1. Retrieved November 2021 from https://www.kaggle.com/fedesoriano/traffic-prediction-dataset (2021)
Haizhong, W., Jia, L., Qian-Yong, C., Daiheng, N., Logistic modeling of the equilibrium speed–density relationship. Transp. Res. Part A: Policy Pract. 45(6), 554–566 (2011)
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The Authors declared that there is no conflict of interest.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Rosayyan, P., Paul, J., Subramaniam, S. et al. Reinforcement Learning Based Control Scheme for Emergency Vehicle Preemption with Edge Computing. Int. J. ITS Res. 21, 48–62 (2023). https://doi.org/10.1007/s13177-022-00334-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13177-022-00334-0