Skip to main content

Advertisement

Log in

An adaptive Q-learning based particle swarm optimization for multi-UAV path planning

  • Optimization
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

In recent times, the path planning of unmanned aerial vehicles (UAVs) in 3D complex flight environments has become a hot topic in the field of UAV technology. Path planning is a crucial process that involves determining the trajectory of the UAV from the point of origin to its destination. However, a number of algorithms proposed for this task have been proven inefficient in this 3D space. In response, this paper proposes the use of an adaptive Q-Learning based particle swarm optimization to tackle the problem. This algorithm introduces the Q-Learning algorithm and designs four states and actions for each particle. Based on the accumulated experience from reinforcement learning, the particles can choose the appropriate action in different states. To evaluate the performance of the AQLPSO algorithm, extensive simulation experiments were conducted. These experiments involved comparing the AQLPSO algorithm with existing algorithms such as PSO, PSO-SA, and RMPSO. The results of the simulations demonstrated that the AQLPSO algorithm outperformed these algorithms in terms of multiple performance metrics. It effectively solved the UAV path planning problem in 3D complex flight environments by reducing the likelihood of falling into local optima, improving efficiency, and achieving faster convergence towards the global optimal solution.

This is a preview of subscription content, log in via an institution to check access.

Access this article

We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

The authors confirm that the data supporting the findings of this study are available within the article [and/or its supplementary materials].

References

  • Albani D, IJsselmuiden J, Haken R, Trianni V ( 2017) Monitoring and mapping with robot swarms for agricultural applications. In: 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1–6 . IEEE

  • Ali N, Kamarudin K, Bakar MAA, Rahiman MHF, Zakaria A, Mamduh SM, Kamarudin LM (2023) 2d lidar based reinforcement learning for multi-target path planning in unknown environment. IEEE Access 11:35541–35555

    Article  Google Scholar 

  • AlShawi IS, Yan L, Pan W, Luo B (2012) Lifetime enhancement in wireless sensor networks using fuzzy approach and a-star algorithm. IEEE Sensors J 12(10):3010–3018

    Article  Google Scholar 

  • Carlucho I, De Paula M, Wang S, Petillot Y, Acosta GG (2018) Adaptive low-level control of autonomous underwater vehicles using deep reinforcement learning. Robot Auton Syst 107:71–86

    Article  Google Scholar 

  • Chen Y-b, Luo G-c, Mei Y-s, Yu J-q, Su X-l (2016) Uav path planning using artificial potential field method updated by optimal control theory. Int J Syst Sci 47(6):1407–1420

    Article  MathSciNet  Google Scholar 

  • Deng L, Chen H, Zhang X, Liu H (2023) Three-dimensional path planning of uav based on improved particle swarm optimization. Mathematics 11(9):1987

    Article  Google Scholar 

  • Duan F, Li X, Zhao Y (2018) Express uav swarm path planning with vnd enhanced memetic algorithm. In: Proceedings of the 2018 International Conference on Computing and Data Engineering, pp. 93–97

  • Guo W, Chen M, Wang L, Mao Y, Wu Q (2017) A survey of biogeography-based optimization. Neural Comput Appl 28:1909–1926

    Article  Google Scholar 

  • Gupta H, Verma OP (2023) A novel hybrid coyote-particle swarm optimization algorithm for three-dimensional constrained trajectory planning of unmanned aerial vehicle. Appl Soft Comput 147:110776

    Article  Google Scholar 

  • Huang H, Jin C (2021) A novel particle swarm optimization algorithm based on reinforcement learning mechanism for auv path planning. Complexity 2021:1–13

    Article  Google Scholar 

  • Huang C, Zhou X, Ran X, Wang J, Chen H, Deng W (2023) Adaptive cylinder vector particle swarm optimization with differential evolution for uav path planning. Eng Appl Artif Intell 121:105942

    Article  Google Scholar 

  • Huuskonen J, Oksanen T (2018) Soil sampling with drones and augmented reality in precision agriculture. Comput Electron Agric 154:25–35

    Article  Google Scholar 

  • Kennedy J, Eberhart R (1995) Particle swarm optimization. In: Proceedings of ICNN’95-international Conference on Neural Networks, vol. 4, pp. 1942–1948. IEEE

  • Kothari M, Postlethwaite I (2013) A probabilistically robust path planning algorithm for uavs using rapidly-exploring random trees. J Intell Robot Syst 71:231–253

    Article  Google Scholar 

  • Kumar P, Garg S, Singh A, Batra S, Kumar N, You I (2018) Mvo-based 2-d path planning scheme for providing quality of service in uav environment. IEEE Internet Things J 5(3):1698–1707

    Article  Google Scholar 

  • Lin S, Liu A, Wang J, Kong X (2023) An intelligence-based hybrid pso-sa for mobile robot path planning in warehouse. J Comput Sci 67:101938

    Article  Google Scholar 

  • Liu J, Wang W, Wang T, Shu Z, Li X (2018) A motif-based rescue mission planning method for uav swarms usingan improved picea. IEEE Access 6:40778–40791

    Article  Google Scholar 

  • Phung MD, Ha QP (2021) Safety-enhanced uav path planning with spherical vector-based particle swarm optimization. Appl Soft Comput 107:107376

    Article  Google Scholar 

  • Rabinovitch J, Lorenz R, Slimko E, Wang K-SC (2021) Scaling sediment mobilization beneath rotorcraft for titan and mars. Aeolian Res 48:100653

    Article  Google Scholar 

  • Radmanesh M, Kumar M (2016) Flight formation of uavs in presence of moving obstacles using fast-dynamic mixed integer linear programming. Aerosp Sci Technol 50:149–160

    Article  Google Scholar 

  • Roudneshin M, Sizkouhi AMM, Aghdam AG (2019) Effective learning algorithms for search and rescue missions in unknown environments. In: WiSEE, pp. 76–80

  • Sreelakshmy K, Gupta H, Verma OP, Kumar K, Ateya AA, Soliman NF (2023) 3d path optimisation of unmanned aerial vehicles using q learning-controlled gwo-aoa. Comput Syst Sci Eng 45(3):2483

    Article  Google Scholar 

  • Sutton RS (1988) Learning to predict by the methods of temporal differences. Mach Learn 3:9–44

    Article  Google Scholar 

  • Wang X, Gursoy MC (2023) Resilient path planning for uavs in data collection under adversarial attacks. IEEE Trans Inf Forens Secur 18:2766–2779

    Article  Google Scholar 

  • Wang G-G, Chu HE, Mirjalili S (2016) Three-dimensional path planning for ucav using an improved bat algorithm. Aerosp Sci Technol 49:231–238

    Article  Google Scholar 

  • Wang Z, Sun G, Zhou K, Zhu L (2023) A parallel particle swarm optimization and enhanced sparrow search algorithm for unmanned aerial vehicle path planning. Heliyon 9(4):e14784

    Article  Google Scholar 

  • Wei M, Wang S, Zheng J, Chen D (2018) Ugv navigation optimization aided by reinforcement learning-based path tracking. IEEE Access 6:57814–57825

    Article  Google Scholar 

  • Wiering MA, Van Otterlo M (2012) Reinforcement learning. Adapt Learn Optim 12(3):729

    Google Scholar 

  • Xia S, Zhang X (2021) Constrained path planning for unmanned aerial vehicle in 3d terrain using modified multi-objective particle swarm optimization. In: Actuators, vol. 10, p. 255 . MDPI

  • Xie R, Meng Z, Zhou Y, Ma Y, Wu Z (2020) Heuristic q-learning based on experience replay for three-dimensional path planning of the unmanned aerial vehicle. Sci Prog 103(1):0036850419879024

    Article  Google Scholar 

  • Yang C-H, Tsai M-H, Kang S-C, Hung C-Y (2018) Uav path planning method for digital terrain model reconstruction-a debris fan example. Autom Constr 93:214–230

    Article  Google Scholar 

  • Yu T, Chang Q (2022) User-guided motion planning with reinforcement learning for human-robot collaboration in smart manufacturing. Expert Syst Appl 209:118291

    Article  Google Scholar 

  • Yu Z, Si Z, Li X, Wang D, Song H (2022) A novel hybrid particle swarm optimization algorithm for path planning of uavs. IEEE Internet Things J 9(22):22547–22558

    Article  Google Scholar 

  • Yu J, Arab A, Yi J, Pei X, Guo X (2022) Hierarchical framework integrating rapidly-exploring random tree with deep reinforcement learning for autonomous vehicle. Appl Intell 53:16473–16486

    Article  Google Scholar 

  • Zhang C, Liu Y, Hu C (2022) Path planning with time windows for multiple uavs based on gray wolf algorithm. Biomimetics 7(4):225

  • Zhao Y, Zheng Z, Liu Y (2018) Survey on computational-intelligence-based uav path planning. Knowl-Based Syst 158:54–64

    Article  Google Scholar 

Download references

Acknowledgements

The work is supported by the Chongqing Natural Science Foundation of China (Grant no. CSTB2022NSCQ-MSX1415).

Funding

This study was funded by Chongqing Natural Science Foundation, China (Grant no. CSTB2022NSCQ-MSX1415).

Author information

Authors and Affiliations

Authors

Contributions

Formulation of overarching research goals and aims: Li Tan; Design of methodology: Hongtao Zhang, Li Tan; Verification of experimental design: Li Tan, Yuzhao Liu, Hongtao Zhang; Designing computer programs: Hongtao Zhang, Ziliang Shang; Data processing and analysis: Hongtao Zhang, Xujie Jiang, Tianli Yuan; Visualization of experimental results: Ziliang Shang, Yuzhao Liu; Writing the initial draft: Hongtao Zhang, Xujie Jiang; Oversight and leadership responsibility for the research activity planning and execution: Li Tan, Hongtao Zhang.

Corresponding author

Correspondence to Li Tan.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Ethics approval

The authors declare that no ethical issues may arise after the publication of this manuscript.

Consent to participate

Informed consent was obtained from all individual participants included in the study.

Consent for publication

All authors agree to publish the paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tan, L., Zhang, H., Liu, Y. et al. An adaptive Q-learning based particle swarm optimization for multi-UAV path planning. Soft Comput 28, 7931–7946 (2024). https://doi.org/10.1007/s00500-024-09691-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-024-09691-2

Keywords

Navigation