Abstract
In recent times, the path planning of unmanned aerial vehicles (UAVs) in 3D complex flight environments has become a hot topic in the field of UAV technology. Path planning is a crucial process that involves determining the trajectory of the UAV from the point of origin to its destination. However, a number of algorithms proposed for this task have been proven inefficient in this 3D space. In response, this paper proposes the use of an adaptive Q-Learning based particle swarm optimization to tackle the problem. This algorithm introduces the Q-Learning algorithm and designs four states and actions for each particle. Based on the accumulated experience from reinforcement learning, the particles can choose the appropriate action in different states. To evaluate the performance of the AQLPSO algorithm, extensive simulation experiments were conducted. These experiments involved comparing the AQLPSO algorithm with existing algorithms such as PSO, PSO-SA, and RMPSO. The results of the simulations demonstrated that the AQLPSO algorithm outperformed these algorithms in terms of multiple performance metrics. It effectively solved the UAV path planning problem in 3D complex flight environments by reducing the likelihood of falling into local optima, improving efficiency, and achieving faster convergence towards the global optimal solution.
Access this article
We’re sorry, something doesn't seem to be working properly.
Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-024-09691-2/MediaObjects/500_2024_9691_Fig1_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-024-09691-2/MediaObjects/500_2024_9691_Fig2_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-024-09691-2/MediaObjects/500_2024_9691_Fig3_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-024-09691-2/MediaObjects/500_2024_9691_Fig4_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-024-09691-2/MediaObjects/500_2024_9691_Fig5_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-024-09691-2/MediaObjects/500_2024_9691_Fig6_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-024-09691-2/MediaObjects/500_2024_9691_Fig7_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-024-09691-2/MediaObjects/500_2024_9691_Fig8_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-024-09691-2/MediaObjects/500_2024_9691_Fig9_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-024-09691-2/MediaObjects/500_2024_9691_Fig10_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-024-09691-2/MediaObjects/500_2024_9691_Fig11_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-024-09691-2/MediaObjects/500_2024_9691_Fig12_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-024-09691-2/MediaObjects/500_2024_9691_Fig13_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-024-09691-2/MediaObjects/500_2024_9691_Fig14_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-024-09691-2/MediaObjects/500_2024_9691_Fig15_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-024-09691-2/MediaObjects/500_2024_9691_Fig16_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-024-09691-2/MediaObjects/500_2024_9691_Fig17_HTML.png)
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data availability
The authors confirm that the data supporting the findings of this study are available within the article [and/or its supplementary materials].
References
Albani D, IJsselmuiden J, Haken R, Trianni V ( 2017) Monitoring and mapping with robot swarms for agricultural applications. In: 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1–6 . IEEE
Ali N, Kamarudin K, Bakar MAA, Rahiman MHF, Zakaria A, Mamduh SM, Kamarudin LM (2023) 2d lidar based reinforcement learning for multi-target path planning in unknown environment. IEEE Access 11:35541–35555
AlShawi IS, Yan L, Pan W, Luo B (2012) Lifetime enhancement in wireless sensor networks using fuzzy approach and a-star algorithm. IEEE Sensors J 12(10):3010–3018
Carlucho I, De Paula M, Wang S, Petillot Y, Acosta GG (2018) Adaptive low-level control of autonomous underwater vehicles using deep reinforcement learning. Robot Auton Syst 107:71–86
Chen Y-b, Luo G-c, Mei Y-s, Yu J-q, Su X-l (2016) Uav path planning using artificial potential field method updated by optimal control theory. Int J Syst Sci 47(6):1407–1420
Deng L, Chen H, Zhang X, Liu H (2023) Three-dimensional path planning of uav based on improved particle swarm optimization. Mathematics 11(9):1987
Duan F, Li X, Zhao Y (2018) Express uav swarm path planning with vnd enhanced memetic algorithm. In: Proceedings of the 2018 International Conference on Computing and Data Engineering, pp. 93–97
Guo W, Chen M, Wang L, Mao Y, Wu Q (2017) A survey of biogeography-based optimization. Neural Comput Appl 28:1909–1926
Gupta H, Verma OP (2023) A novel hybrid coyote-particle swarm optimization algorithm for three-dimensional constrained trajectory planning of unmanned aerial vehicle. Appl Soft Comput 147:110776
Huang H, Jin C (2021) A novel particle swarm optimization algorithm based on reinforcement learning mechanism for auv path planning. Complexity 2021:1–13
Huang C, Zhou X, Ran X, Wang J, Chen H, Deng W (2023) Adaptive cylinder vector particle swarm optimization with differential evolution for uav path planning. Eng Appl Artif Intell 121:105942
Huuskonen J, Oksanen T (2018) Soil sampling with drones and augmented reality in precision agriculture. Comput Electron Agric 154:25–35
Kennedy J, Eberhart R (1995) Particle swarm optimization. In: Proceedings of ICNN’95-international Conference on Neural Networks, vol. 4, pp. 1942–1948. IEEE
Kothari M, Postlethwaite I (2013) A probabilistically robust path planning algorithm for uavs using rapidly-exploring random trees. J Intell Robot Syst 71:231–253
Kumar P, Garg S, Singh A, Batra S, Kumar N, You I (2018) Mvo-based 2-d path planning scheme for providing quality of service in uav environment. IEEE Internet Things J 5(3):1698–1707
Lin S, Liu A, Wang J, Kong X (2023) An intelligence-based hybrid pso-sa for mobile robot path planning in warehouse. J Comput Sci 67:101938
Liu J, Wang W, Wang T, Shu Z, Li X (2018) A motif-based rescue mission planning method for uav swarms usingan improved picea. IEEE Access 6:40778–40791
Phung MD, Ha QP (2021) Safety-enhanced uav path planning with spherical vector-based particle swarm optimization. Appl Soft Comput 107:107376
Rabinovitch J, Lorenz R, Slimko E, Wang K-SC (2021) Scaling sediment mobilization beneath rotorcraft for titan and mars. Aeolian Res 48:100653
Radmanesh M, Kumar M (2016) Flight formation of uavs in presence of moving obstacles using fast-dynamic mixed integer linear programming. Aerosp Sci Technol 50:149–160
Roudneshin M, Sizkouhi AMM, Aghdam AG (2019) Effective learning algorithms for search and rescue missions in unknown environments. In: WiSEE, pp. 76–80
Sreelakshmy K, Gupta H, Verma OP, Kumar K, Ateya AA, Soliman NF (2023) 3d path optimisation of unmanned aerial vehicles using q learning-controlled gwo-aoa. Comput Syst Sci Eng 45(3):2483
Sutton RS (1988) Learning to predict by the methods of temporal differences. Mach Learn 3:9–44
Wang X, Gursoy MC (2023) Resilient path planning for uavs in data collection under adversarial attacks. IEEE Trans Inf Forens Secur 18:2766–2779
Wang G-G, Chu HE, Mirjalili S (2016) Three-dimensional path planning for ucav using an improved bat algorithm. Aerosp Sci Technol 49:231–238
Wang Z, Sun G, Zhou K, Zhu L (2023) A parallel particle swarm optimization and enhanced sparrow search algorithm for unmanned aerial vehicle path planning. Heliyon 9(4):e14784
Wei M, Wang S, Zheng J, Chen D (2018) Ugv navigation optimization aided by reinforcement learning-based path tracking. IEEE Access 6:57814–57825
Wiering MA, Van Otterlo M (2012) Reinforcement learning. Adapt Learn Optim 12(3):729
Xia S, Zhang X (2021) Constrained path planning for unmanned aerial vehicle in 3d terrain using modified multi-objective particle swarm optimization. In: Actuators, vol. 10, p. 255 . MDPI
Xie R, Meng Z, Zhou Y, Ma Y, Wu Z (2020) Heuristic q-learning based on experience replay for three-dimensional path planning of the unmanned aerial vehicle. Sci Prog 103(1):0036850419879024
Yang C-H, Tsai M-H, Kang S-C, Hung C-Y (2018) Uav path planning method for digital terrain model reconstruction-a debris fan example. Autom Constr 93:214–230
Yu T, Chang Q (2022) User-guided motion planning with reinforcement learning for human-robot collaboration in smart manufacturing. Expert Syst Appl 209:118291
Yu Z, Si Z, Li X, Wang D, Song H (2022) A novel hybrid particle swarm optimization algorithm for path planning of uavs. IEEE Internet Things J 9(22):22547–22558
Yu J, Arab A, Yi J, Pei X, Guo X (2022) Hierarchical framework integrating rapidly-exploring random tree with deep reinforcement learning for autonomous vehicle. Appl Intell 53:16473–16486
Zhang C, Liu Y, Hu C (2022) Path planning with time windows for multiple uavs based on gray wolf algorithm. Biomimetics 7(4):225
Zhao Y, Zheng Z, Liu Y (2018) Survey on computational-intelligence-based uav path planning. Knowl-Based Syst 158:54–64
Acknowledgements
The work is supported by the Chongqing Natural Science Foundation of China (Grant no. CSTB2022NSCQ-MSX1415).
Funding
This study was funded by Chongqing Natural Science Foundation, China (Grant no. CSTB2022NSCQ-MSX1415).
Author information
Authors and Affiliations
Contributions
Formulation of overarching research goals and aims: Li Tan; Design of methodology: Hongtao Zhang, Li Tan; Verification of experimental design: Li Tan, Yuzhao Liu, Hongtao Zhang; Designing computer programs: Hongtao Zhang, Ziliang Shang; Data processing and analysis: Hongtao Zhang, Xujie Jiang, Tianli Yuan; Visualization of experimental results: Ziliang Shang, Yuzhao Liu; Writing the initial draft: Hongtao Zhang, Xujie Jiang; Oversight and leadership responsibility for the research activity planning and execution: Li Tan, Hongtao Zhang.
Corresponding author
Ethics declarations
Conflict of interest
The authors have no competing interests to declare that are relevant to the content of this article.
Ethics approval
The authors declare that no ethical issues may arise after the publication of this manuscript.
Consent to participate
Informed consent was obtained from all individual participants included in the study.
Consent for publication
All authors agree to publish the paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Tan, L., Zhang, H., Liu, Y. et al. An adaptive Q-learning based particle swarm optimization for multi-UAV path planning. Soft Comput 28, 7931–7946 (2024). https://doi.org/10.1007/s00500-024-09691-2
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-024-09691-2