skip to main content
10.1145/3397536.3422204acmconferencesArticle/Chapter ViewAbstractPublication PagesgisConference Proceedingsconference-collections
poster

Ambulance Dispatch via Deep Reinforcement Learning

Published: 13 November 2020 Publication History

Abstract

In this paper, we solve the ambulance dispatch problem with a reinforcement learning oriented strategy. The ambulance dispatch problem is defined as deciding which ambulance to pick up which patient. Traditional studies on ambulance dispatch mainly focus on predefined protocols and are verified on simple simulation data, which are not flexible enough when facing the dynamically changing real-world cases. In this paper, we propose an efficient ambulance dispatch method based on the reinforcement learning framework, i.e., Multi-Agent Q-Network with Experience Replay(MAQR). Specifically, we firstly reformulate the ambulance dispatch problem with a multi-agent reinforcement learning framework, and then design the state, action, and reward function correspondingly for the framework. Thirdly, we design a simulator that controls ambulance status, generates patient requests and interacts with ambulances. Finally, we design extensive experiments to demonstrate the superiority of the proposed method.

References

[1]
Ester Alessandrini, Stefano Zauli Sajani, Fabiana Scotto, Rossella Miglio, Stefano Marchesi, and Paolo Lauriola. 2011. Emergency ambulance dispatches and apparent temperature: a time series analysis in Emilia-Romagna, Italy. Environmental research 111, 8 (2011), 1192--1200.
[2]
Luce Brotcorne, Gilbert Laporte, and Frederic Semet. 2003. Ambulance location and relocation models. European journal of operational research 147, 3 (2003), 451--463.
[3]
Timothy A Carnes, Shane G Henderson, David B Shmoys, Mahvareh Ahghari, and Russell D MacDonald. 2013. Mathematical programming guides air-ambulance routing at ornge. Interfaces 43, 3 (2013), 232--239.
[4]
Richard Church and Charles ReVelle. 1974. The maximal covering location problem. In Papers of the Regional Science Association, Vol. 32. Springer-Verlag, 101--118.
[5]
Wei Fan, Kunpeng Liu, Hao Liu, Pengyang Wang, Yong Ge, and Yanjie Fu. 2020. AutoFS: Automated Feature Selection via Diversity-aware Interactive Reinforcement Learning. arXiv preprint arXiv:2008.12001 (2020).
[6]
Michel Gendreau, Gilbert Laporte, and Frédéric Semet. 2001. A dynamic model and parallel tabu search heuristic for real-time ambulance relocation. Parallel computing 27, 12 (2001), 1641--1653.
[7]
Jeffrey B Goldberg. 2004. Operations research models for the deployment of emergency services vehicles. EMS management Journal 1, 1 (2004), 20--39.
[8]
Jared Hayes, Antoni Moore, George Benwell, and BL William Wong. 2004. Ambulance dispatch complexity and dispatcher decision strategies: Implications for interface design. In Asia-Pacific Conference on Computer Human Interaction. Springer, 589--593.
[9]
HL Liao, QH Wu, and L Jiang. 2010. Multi-objective optimization by reinforcement learning for power system dispatch and voltage stability. In Innovative Smart Grid Technologies Conference Europe (ISGT Europe), 2010 IEEE PES. IEEE, 1--8.
[10]
Kaixiang Lin, Renyu Zhao, Zhe Xu, and Jiayu Zhou. 2018. Efficient Large-Scale Fleet Management via Multi-Agent Deep Reinforcement Learning. arXiv preprint arXiv:1802.06444 (2018).
[11]
Kunpeng Liu, Yanjie Fu, Pengfei Wang, Le Wu, Rui Bo, and Xiaolin Li. 2019. Automating Feature Subspace Exploration via Multi-Agent Reinforcement Learning. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 207--215.
[12]
Kunpeng Liu, Pengyang Wang, Jiawei Zhang, Yanjie Fu, and Sajal K Das. 2018. Modeling the Interaction Coupling of Multi-View Spatiotemporal Contexts for Destination Prediction. In Proceedings of the 2018 SIAM International Conference on Data Mining. SIAM, 171--179.
[13]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. nature 518, 7540 (2015), 529--533.
[14]
Noraimi Azlin Mohd Nordin, Zati Aqmar Zaharudin, Mohd Azdi Maasar, and Nor Amalina Nordin. 2012. Finding shortest path of the ambulance routing: Interface of A algorithm using C# programming. In Humanities, Science and Engineering Research (SHUSER), 2012 IEEE Symposium on. IEEE, 1569--1573.
[15]
Imtiyaz Pasha. 2006. Ambulance management system using GIS. Universitetsbibliotek.
[16]
Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. 2017. Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games. arXiv preprint arXiv: 1703.10069 (2017).
[17]
Martina Petralli, Marco Morabito, Lorenzo Cecchi, Alfonso Crisci, and Simone Orlandini. 2012. Urban morbidity in summer: ambulance dispatch data, periodicity and weather. Central European Journal of Medicine 7, 6 (2012), 775--782.
[18]
John F Repede and John J Bernardo. 1994. Developing and validating a decision support system for locating emergency medical vehicles in Louisville, Kentucky. European journal of operational research 75, 3 (1994), 567--581.
[19]
Milos Stankovic. 2016. Multi-agent reinforcement learning. In Neural Networks and Applications (NEUREL), 2016 13th Symposium on. IEEE, 1--1.
[20]
Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. 2017. Multiagent cooperation and competition with deep reinforcement learning. PloS one 12, 4 (2017), e0172395.
[21]
TC Van Barneveld, S Bhulai, and RD Van der Mei. 2017. A dynamic ambulance management model for rural areas. Health care management science 20, 2 (2017), 165--186.
[22]
Hua Wei, Guanjie Zheng, Huaxiu Yao, and Zhenhui Li. 2018. IntelliLight: A Reinforcement Learning Approach for Intelligent Traffic Light Control. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2496--2505.
[23]
Andres Weintraub, Julio Aboud, C Fernandez, Gilbert Laporte, and E Ramirez. 1999. An emergency vehicle dispatching system for an electric utility in Chile. Journal of the Operational Research Society 50, 7 (1999), 690--696.
[24]
Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. 2018. Mean Field Multi-Agent Reinforcement Learning. arXiv preprint arXiv:1802.05438 (2018).
[25]
Lianmin Zheng, Jiacheng Yang, Han Cai, Weinan Zhang, Jun Wang, and Yong Yu. 2017. MAgent: A Many-Agent Reinforcement Learning Platform for Artificial Collective Intelligence. arXiv preprint arXiv:1712.00600 (2017).

Cited By

View all
  • (2024)Multi-Objective Deep Reinforcement Learning for Variable Speed Limit ControlProceedings of the 2024 16th International Conference on Machine Learning and Computing10.1145/3651671.3651719(621-627)Online publication date: 2-Feb-2024
  • (2024)A Survey of Machine Learning Innovations in Ambulance Services: Allocation, Routing, and Demand EstimationIEEE Open Journal of Intelligent Transportation Systems10.1109/OJITS.2024.3514871(1-1)Online publication date: 2024
  • (2023)Q-learning Based Simulation Tool for Studying Effectiveness of Dynamic Application of Fertilizer on Crop ProductivityProceedings of the 11th ACM SIGSPATIAL International Workshop on Analytics for Big Geospatial Data10.1145/3615833.3628591(13-22)Online publication date: 13-Nov-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGSPATIAL '20: Proceedings of the 28th International Conference on Advances in Geographic Information Systems
November 2020
687 pages
ISBN:9781450380195
DOI:10.1145/3397536
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 November 2020

Check for updates

Author Tags

  1. ambulance dispatch
  2. experience replay
  3. reinforcement learning

Qualifiers

  • Poster
  • Research
  • Refereed limited

Conference

SIGSPATIAL '20
Sponsor:

Acceptance Rates

Overall Acceptance Rate 257 of 1,238 submissions, 21%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)74
  • Downloads (Last 6 weeks)7
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Multi-Objective Deep Reinforcement Learning for Variable Speed Limit ControlProceedings of the 2024 16th International Conference on Machine Learning and Computing10.1145/3651671.3651719(621-627)Online publication date: 2-Feb-2024
  • (2024)A Survey of Machine Learning Innovations in Ambulance Services: Allocation, Routing, and Demand EstimationIEEE Open Journal of Intelligent Transportation Systems10.1109/OJITS.2024.3514871(1-1)Online publication date: 2024
  • (2023)Q-learning Based Simulation Tool for Studying Effectiveness of Dynamic Application of Fertilizer on Crop ProductivityProceedings of the 11th ACM SIGSPATIAL International Workshop on Analytics for Big Geospatial Data10.1145/3615833.3628591(13-22)Online publication date: 13-Nov-2023
  • (2023)Equity-Aware Cross-Graph Interactive Reinforcement Learning for Bike Station Network ExpansionProceedings of the 31st ACM International Conference on Advances in Geographic Information Systems10.1145/3589132.3625588(1-12)Online publication date: 13-Nov-2023
  • (2023)Learning Emergency Medical Dispatch Policies Via Genetic ProgrammingProceedings of the Genetic and Evolutionary Computation Conference10.1145/3583131.3590434(1409-1417)Online publication date: 15-Jul-2023
  • (2023)FairCod: A Fairness-aware Concurrent Dispatch System for Large-scale Instant Delivery ServicesProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599824(4229-4238)Online publication date: 6-Aug-2023
  • (2023)Online POI Recommendation: Learning Dynamic Geo-Human Interactions in StreamsIEEE Transactions on Big Data10.1109/TBDATA.2022.32151349:3(832-844)Online publication date: 1-Jun-2023
  • (2023)Smart crisis management system for road accidents based on Modified Convolutional Neural Networks–Particle Swarm Optimization hybrid algorithm10.1016/bs.adcom.2023.07.002Online publication date: 2023
  • (2023)Teaching old tools new tricks—preparing emergency medicine for the impact of machine learning-based risk prediction modelsCanadian Journal of Emergency Medicine10.1007/s43678-023-00480-825:5(365-369)Online publication date: 18-Mar-2023
  • (2023)Repositioning Fleet Vehicles: A Learning PipelineLearning and Intelligent Optimization10.1007/978-3-031-44505-7_21(301-317)Online publication date: 25-Oct-2023
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media