Abstract
This paper proposes a design scheme for a whole morphing strategy based on the reinforcement learning (RL) method. A novel morphing aircraft is designed, and its nonlinear dynamic equations are established based on the calculated aerodynamic data. Further, a soft actor critic (SAC) approach is utilized to design the scheme, whose structure consists of the environment, the agent, and the reward function. In the environment design part, the incremental backstepping approach is employed to design the morphing aircraft controller. The safety and feasibility of deployment are verified. In the agent design part, in addition to using the entropy regularization RL algorithm, the generalization ability of the agent is enhanced in three ways: adding environmental noise, adding control command randomness, and adding output momentum terms. For the reward function, a structure with dynamic and steady-state performance is designed to accurately describe the aircraft dynamics. Finally, the designed SAC strategy is verified under the acceleration and deceleration tasks and compared with a GA and PPO strategy. Simulation results validate the effectiveness and superiority of the designed SAC scheme.













Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Tsushima N, Tamayama M (2019) Recent researches on morphing aircraft technologies in Japan and other countries. Mech Eng Rev 6(2):19–00197–19–00197. https://doi.org/10.1299/mer.19-00197
Kölbl M, Ermanni P (2022) Structural design and analysis of an anisotropic, bi-axially morphing skin concept. Aerosp Sci Technol 120. https://doi.org/10.1016/j.ast.2021.107292
Hao F, Tang T, Gao Y, Li Y, Yi S, Lu J (2021) Continuous morphing trailing-edge wing concept based on multi-stable nanomaterial. Chin J Aeronaut 34(7):219–231. https://doi.org/10.1016/j.cja.2020.03.041
Kammegne MJT, Botez RM, Grigorie LT, Mamou M, Mébarki Y (2017) Proportional fuzzy feed-forward architecture control validation by wind tunnel tests of a morphing wing. Chin J Aeronaut 30(2):561–576. https://doi.org/10.1016/j.cja.2017.02.001
Xu W, Li Y, Pei B, Yu Z (2022) Coordinated intelligent control of the flight control system and shape change of variable sweep morphing aircraft based on dueling-DQN. Aerosp Sci Technol 130. https://doi.org/10.1016/j.ast.2022.107898
Li D et al (2018) A review of modelling and analysis of morphing wings. Progress Aerospace Sci 100:46–62. https://doi.org/10.1016/j.paerosci.2018.06.002
Wu M, Xiao T, Ang H, Li H (2018) Optimal flight planning for a Z-shaped morphing-wing solar-powered unmanned aerial vehicle. J Guid Control Dyn 41(2):497–505. https://doi.org/10.2514/1.G003000
Xu W, Li Y, Lv M, Pei B (2022) Modeling and switching adaptive control for nonlinear morphing aircraft considering actuator dynamics. Aerosp Sci Technol 122. https://doi.org/10.1016/j.ast.2022.107349
Qiao F, Shi J, Qu X, Lyu Y (2022) Adaptive Back-stepping neural control for an embedded and Tiltable V-tail morphing aircraft. Int J Control Autom Syst 20(2):678–690. https://doi.org/10.1007/s12555-020-0694-0
Wu Z, Lu J, Shi J, Liu Y, Zhou Q (2017) Robust adaptive neural control of morphing aircraft with prescribed performance. Math Probl Eng 2017. https://doi.org/10.1155/2017/1401427
Hammerton JR, Su W, Zhu G, Swei SSM (2018) Optimum distributed wing shaping and control loads for highly flexible aircraft. Aerosp Sci Technol 79:255–265. https://doi.org/10.1016/j.ast.2018.05.045
Gu X, Yang K, Wu M, Zhang Y, Zhu J, Zhang W (2021) Integrated optimization design of smart morphing wing for accurate shape control. Chin J Aeronaut 34(1):135–147. https://doi.org/10.1016/j.cja.2020.08.048
Mansell JR, Grant MJ (2018) Adaptive continuation strategy for indirect hypersonic trajectory optimization. J Spacecr Rocket 55(4):818–828. https://doi.org/10.2514/1.A34013
Li W, Wang W, Huang X, Zhang S, Li C (2021) Roll control of morphing aircraft with synthetic jet actuators at a high angle of attack. Appl Sci 11(2):1–12. https://doi.org/10.3390/app11020505
Themistokleous C, Markatos NG, Prospathopoulos J, Riziotis V, Sieros G, Papadakis G. A high-lift optimization methodology for the design of leading and trailing edges on morphing wings. Appl Sci 11(6). https://doi.org/10.3390/app11062822
Gong C, Ma BF (2019) Shape optimization and sensitivity analysis of a morphing-wing aircraft. Int J Aeronaut Sp Sci 20(1):57–69. https://doi.org/10.1007/s42405-018-0110-7
Dai P, Yan B, Liu R, Liu S, Wang M (2021) Integrated morphing strategy and trajectory optimization of a morphing Waverider and its online implementation based on the neural network. IEEE Access 9:59383–59393. https://doi.org/10.1109/ACCESS.2021.3073207
Silver D et al (2017) Mastering the game of go without human knowledge. Nature 550(7676):354–359. https://doi.org/10.1038/nature24270
Muthusamy PK, Garratt M, Pota HR, Muthusamy R (2021) Realtime adaptive intelligent control system for quadcopter UAV with payload uncertainties. IEEE Trans Ind Electron. https://doi.org/10.1109/TIE.2021.3055170
Jardine PT, Givigi SN, Yousefi S (2021) Leveraging data engineering to improve unmanned aerial vehicle control design. IEEE Syst J 15(2):2595–2606. https://doi.org/10.1109/JSYST.2020.3003203
Lin X, Liu J, Yu Y, Sun C (2020) Event-triggered reinforcement learning control for the quadrotor UAV with actuator saturation. Neurocomputing 415:135–145. https://doi.org/10.1016/j.neucom.2020.07.042
Hajijamali Arani A, Azari MM, Hu P, Zhu Y, Yanikomeroglu H, Safavi-Naeini S (2022) Reinforcement learning for energy-efficient trajectory design of UAVs. IEEE Internet Things J 9(11):9060–9070. https://doi.org/10.1109/JIOT.2021.3118322
Liu Y, Wang H, Wu T, Lun Y, Fan J, Wu J (2022) Attitude control for hypersonic reentry vehicles: An efficient deep reinforcement learning method. Appl Soft Comput 123. https://doi.org/10.1016/j.asoc.2022.108865
Lu K, Fu Q, Cao R, Peng J, Wang Q (2022) Asymmetric Airfoil Morphing via Deep Reinforcement Learning. Biomimetics 7(4):188. https://doi.org/10.3390/biomimetics7040188
Haughn KPT, Gamble LL, Inman DJ (2022) Deep reinforcement learning achieves multifunctional morphing airfoil control. J Compos Mater. https://doi.org/10.1177/00219983221137644
Xu D, Hui Z, Liu Y, Chen G (2019) Morphing control of a new bionic morphing UAV with deep reinforcement learning. Aerosp Sci Technol 92:232–243. https://doi.org/10.1016/j.ast.2019.05.058
Che H, Chen J, Wang Y, Wang J, Luo Y (2022) Data-driven model-free adaptive attitude control for morphing vehicles. IET Control Theory Appl 16(16):1696–1707. https://doi.org/10.1049/cth2.12335
Seigler TM, Neal DA (2009) Analysis of transition stability for morphing aircraft. J Guid Control Dyn 32(6):1947–1953. https://doi.org/10.2514/1.44108
Haarnoja T, Zhou A, Abbeel P, Levine S (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: Dy J, Krause A (eds) Proceedings of the 35th International Conference on Machine Learning, vol 80 of Proceedings of Machine Learning Research. PMLR, pp 1861–1870. https://proceedings.mlr.press/v80/haarnoja18b.html
Acknowledgements
The authors would like to express their gratitude to the Shaanxi Province Key Laboratory of Flight Control and Simulation Technology for supporting this research. This research work is funded by the National Natural Science Foundation of China (No. 62073266) and the Aeronautical Science Foundation of China (No. 201905053003).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing of interests
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
The authors declare the following financial interests/personal relationships which may be considered as potential competing interests.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Ming, ., Liu, X., Li, Y. et al. Morphing aircraft acceleration and deceleration task morphing strategy using a reinforcement learning method. Appl Intell 53, 26637–26654 (2023). https://doi.org/10.1007/s10489-023-04876-y
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-023-04876-y