Abstract
The Deep Q-Network (DQN) is a deep reinforcement learning method using Convolution Neural Network (CNN) as a function approximation of Q-values in the Q-learning algorithm. The DQN combines the neural fitting Q-iteration and experience replay, shares the hidden layer of the action value function for each action pattern and can stabilize learning even with nonlinear functions such as CNN. However, there are some points where learning is difficult to progress for problems with complex operations and rewards, or problems where it takes a long time to obtain a reward. In this paper, we present a DQN-based Autonomous Aerial Vehicle (AAV) mobility control method and show the performance evaluation results for a corner environment scenario. The performance evaluation results show that the proposed method can reach the destination and can decrease movement fluctuations in a corner environment.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Stöcker, C., et al.: Review of the current state of UAV regulations. Remote Sens. 9(5), 1–26 (2017)
Popović, M., et al.: An informative path planning framework for UAV-based terrain monitoring. Auton. Robot. 44, 889–911 (2020). https://doi.org/10.1007/s10514-020-09903-2
Nguyen, H., et al.: LAVAPilot: lightweight UAV trajectory planner with situational awareness for embedded autonomy to track and locate radio-tags, pp. 1–8. arXiv:2007.15860 (2020)
Saito, N., Oda, T., Hirata, A., Hirota, Y., Hirota, M., Katayama, K.: Design and implementation of a DQN based AAV. In: Barolli, L., Takizawa, M., Enokido, T., Chen, H.-C., Matsuo, K. (eds.) BWCCA 2020. LNNS, vol. 159, pp. 321–329. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-61108-8_32
Oda, T., et al.: Design and implementation of a simulation system based on deep Q-network for mobile actor node control in wireless sensor and actor networks. In: Proceedings of The 31th IEEE International Conference on Advanced Information Networking and Applications Workshops (IEEE AINA-2017), pp. 195–200 (2017)
Oda, T., Elmazi, D., Cuka, M., Kulla, E., Ikeda, M., Barolli, L.: Performance evaluation of a deep Q-network based simulation system for actor node mobility control in wireless sensor and actor networks considering three-dimensional environment. In: Barolli, L., Woungang, I., Hussain, O.K. (eds.) INCoS 2017. LNDECT, vol. 8, pp. 41–52. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-65636-6_4
Oda, T., Kulla, E., Katayama, K., Ikeda, M., Barolli, L.: A deep Q-network based simulation system for actor node mobility control in WSANs considering three-dimensional environment: a comparison study for normal and uniform distributions. In: Barolli, L., Javaid, N., Ikeda, M., Takizawa, M. (eds.) CISIS 2018. AISC, vol. 772, pp. 842–852. Springer, Cham (2019). https://doi.org/10.1007/978-3-319-93659-8_77
Saito, N., Oda, T., Hirata, A., Nagai, Y., Hirota, M., Katayama, K.: Proposal and evaluation of a Tabu list based DQN for AAV mobility. In: Barolli, L., Natwichai, J., Enokido, T. (eds.) EIDWT 2021. LNDECT, vol. 65, pp. 189–200. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-70639-5_18
Moulton, J., et al.: An autonomous surface vehicle for long term operations. In: Proceedings of MTS/IEEE OCEANS, pp. 1–10 (2018)
Oda, T., Ueda, C., Ozaki, R., Katayama, K.: Design of a deep Q-network based simulation system for actuation decision in ambient intelligence. In: Barolli, L., Takizawa, M., Xhafa, F., Enokido, T. (eds.) WAINA 2019. AISC, vol. 927, pp. 362–370. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-15035-8_34
Oda, T., et al.: Design and implementation of an IoT-based e-learning testbed. Int. J. Web Grid Serv. 13(2), 228–241 (2017)
Hirota, Y., Oda, T., Saito, N., Hirata, A., Hirota, M., Katatama, K.: Proposal and experimental results of an ambient intelligence for training on soldering iron holding. In: Barolli, L., Takizawa, M., Enokido, T., Chen, H.-C., Matsuo, K. (eds.) BWCCA 2020. LNNS, vol. 159, pp. 444–453. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-61108-8_44
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)
De Raedt, L.: Statistical relational learning: an inductive logic programming perspective. In: Gama, J., Camacho, R., Brazdil, P.B., Jorge, A.M., Torgo, L. (eds.) ECML 2005. LNCS (LNAI), vol. 3720, pp. 3–5. Springer, Heidelberg (2005). https://doi.org/10.1007/11564096_3
Lin, L.J.: Reinforcement learning for robots using neural networks. Proceedings of technical report, DTIC Document (1993)
Lange, S., Riedmiller, M.: Deep auto-encoder neural networks in reinforcement learning. In: Proceedings of the International Joint Conference on Neural Networks (IJCNN-2010), pp. 1–8 (2010)
Saito, N., Oda, T., Hirata, A., Yukawa, C., Hirota, M., Barolli, L.: Performance evaluation of a DQN-based autonomous aerial vehicle mobility control method in an indoor single-path environment with a staircase. In: Barolli, L., Kulla, E., Ikeda, M. (eds.) EIDWT 2022. LNDECT, vol. 118, pp. 417–429. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-95903-6_44
Saito, N., et al.: Performance evaluation of a DQN-based autonomous aerial vehicle mobility control method in corner environment. In: Barolli, L., Hussain, F., Enokido, T. (eds.) AINA 2022. LNNS, vol. 450, pp. 361–372. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-99587-4_31
Saito, N., et al.: A Tabu list strategy based DQN for AAV mobility in indoor single-path environment: implementation and performance evaluation. Internet Things 14, 100394 (2021)
Saito, N., Oda, T., Hirata, A., Yukawa, C., Kulla, E., Barolli, L.: A LiDAR based mobile area decision method for TLS-DQN: improving control for AAV mobility. In: Barolli, L. (ed.) 3PGCIC 2021. LNNS, vol. 343, pp. 30–42. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-89899-1_4
Saito, N., Oda, T., Hirata, A., Toyoshima, K., Hirota, M., Barolli, L.: Simulation results of a DQN based AAV testbed in corner environment: a comparison study for normal DQN and TLS-DQN. In: Barolli, L., Yim, K., Chen, H.-C. (eds.) IMIS 2021. LNNS, vol. 279, pp. 156–167. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-79728-7_16
Glorot, X., et al.: Deep sparse rectifier neural networks. In: Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS-2011), pp. 315–323 (2011)
Glover, F.: Tabu search - part I. ORSA J. Comput. 1(3), 190–206 (1989)
Acknowledgement
This work was supported by JSPS KAKENHI Grant Number JP20K19793 and Grant for Promotion of OUS Research Project (OUS-RP-20-3).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Saito, N. et al. (2022). A Movement Adjustment Method for DQN-Based Autonomous Aerial Vehicle Mobility: Performance Evaluation of AAV Mobility Control Method in Corner Environment. In: Barolli, L., Miwa, H. (eds) Advances in Intelligent Networking and Collaborative Systems. INCoS 2022. Lecture Notes in Networks and Systems, vol 527. Springer, Cham. https://doi.org/10.1007/978-3-031-14627-5_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-14627-5_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-14626-8
Online ISBN: 978-3-031-14627-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)