Skip to main content
SpringerLink
Log in

A deep Q network assisted method for underwater gliders standoff tracking to the static target

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Underwater gliders lack the necessary navigation equipment and have low control performance, which deteriorate the autonomy and efficiency of the sampling. The underwater gliders standoff tracking based on the Lyapunov guidance vector fields is introduced in this work to enhance the autonomy of gliders in observing the potential static targets. To avoid designing complex control processes, we convert the standoff tracking into a Markovian decision process and introduce reinforcement learning methods to solve the task. Also, to trade-off the fast training and achieving acceptable results, we design a control framework that integrates classical controller and reinforcement learning. The simulations show that the proposed framework outperform than the comparison method. This work can provide a new pattern for the sampling control of gliders. The proposed method combining reinforcement learning with classical controller can provide a reference for other applications of reinforcement learning.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Data Availability Statement

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

References

  1. Petritoli E, Cagnetti M, Leccese F (2020) Simulation of autonomous underwater vehicles (auvs) swarm diffusion. Sensors 20(17):4950

    Article  Google Scholar 

  2. Petritoli E, Leccese F, Cagnetti M (2020) Underwater gliders: mission profiles and utilisation strategies in the mediterranean sea. In: 2019 IMEKO TC19 international workshop on metrology for the sea: learning to measure sea health parameters. MetroSea 2019, pp 254–258

  3. Zhang Y, Zhang Z, Quan Z, Liu G-L (2021) Hydrodynamic performance and calculation of lift-drag ratio on underwater glider. J Mar Sci Technol 26(1):16–23

    Article  Google Scholar 

  4. Zang W, Yao P, Song D (2022) Standoff tracking control of underwater glider to moving target. Appl Math Modell 102:1–20

    Article  MathSciNet  Google Scholar 

  5. Du X, Zhang X (2020) Influence of ocean currents on the stability of underwater glider self-mooring motion with a cable. Nonlinear Dyn 99(3):2291–2317

    Article  Google Scholar 

  6. Benoit-Bird KJ, Patrick Welch T, Waluk CM et al (2018) Equipping an underwater glider with a new echosounder to explore ocean ecosystems. Limnol. Oceanogr. Methods 16(11):734–749

    Article  Google Scholar 

  7. Chen X, Kong Y, Fang X, Wu Q (2013) A fast two-stage aco algorithm for robotic path planning. Neural Comput Appl 22(2):313–319

    Article  Google Scholar 

  8. Petritoli E, Leccese F (2018) High accuracy attitude and navigation system for an autonomous underwater vehicle (auv). Acta Imeko 7(2):3–9

    Article  Google Scholar 

  9. Wu H, Niu W, Wang S, Yan S (2021) An optimization method for control parameters of underwater gliders considering energy consumption and motion accuracy. Appl Math Modell 90:1099–1119

    Article  MathSciNet  MATH  Google Scholar 

  10. Shu Y, Chen J, Li S, Wang Q, Yu J, Wang D (2019) Field-observation for an anticyclonic mesoscale eddy consisted of twelve gliders and sixty-two expendable probes in the northern south china sea during summer 2017. Sci China Earth Sci 62(2):451–458

    Article  Google Scholar 

  11. Qiu C, Mao H, Liu H, Xie Q, Yu J, Su D, Ouyang J, Lian S (2019) Deformation of a warm eddy in the northern south china sea. J Geophys Res Oceans 124(8):5551–5564

    Article  Google Scholar 

  12. Yao P, Wang H, Su Z (2016) Cooperative path planning with applications to target tracking and obstacle avoidance for multi-uavs. Aerosp Sci Technol 54:10–22

    Article  Google Scholar 

  13. Liblik T, Karstensen J, Testor P, Alenius P, Hayes D, Ruiz S, Heywood K, Pouliquen S, Mortier L, Mauri E (2016) Potential for an underwater glider component as part of the global ocean observing system. Methods Oceanogr 17:50–82

    Article  Google Scholar 

  14. Lawrence D (2003) Lyapunov vector fields for uav flock coordination. In: 2nd AIAA “Unmanned Unlimited” Conf. and Workshop & Exhibit, p 6575

  15. Wei X, Yao P, Xie Z (2020) Comprehensive optimization of energy storage and standoff tracking for solarpowered UAV. IEEE Syst J 14(4):5133–5143

    Article  Google Scholar 

  16. Liang Y, Jia Y, Du J, Zhang J (2015) Vector field guidance for three-dimensional curved path following with fixed-wing uavs. In: 2015 American control conference (ACC). IEEE, pp 1187–1192

  17. Zhang S, Yu J, Zhang A, Zhang F (2013) Spiraling motion of underwater gliders: modeling, analysis, and experimental results. Ocean Eng 60:1–13

    Article  Google Scholar 

  18. Fu J, Zhang J, She Z, Ovur SE, Li W, Qi W, Su H, Ferrigno G, De Momi E (2021) Whole-body spatial teleoperation control of a hexapod robot in unstructured environment. In: 2021 6th IEEE international conference on advanced robotics and mechatronics (ICARM). IEEE, pp 93–98

  19. Cao F (2020) Pid controller optimized by genetic algorithm for direct-drive servo system. Neural Comput Appl 32(1):23–30

    Article  Google Scholar 

  20. Sang H, Zhou Y, Sun X, Yang S (2018) Heading tracking control with an adaptive hybrid control for under actuated underwater glider. ISA Trans 80:554–563

    Article  Google Scholar 

  21. Zhou S, Zhou Y, Xu Z, Chang W, Cheng Y (2019) The landing safety prediction model by integrating pattern recognition and markov chain with flight data. Neural Comput Appl 31(1):147–159

    Article  Google Scholar 

  22. Sutton RS, Barto AG (2018) Reinforcement learning: an introduction. MIT Press, Cambridge

    MATH  Google Scholar 

  23. Sun Y, Ran X, Zhang G, Xu H, Wang X (2020) Auv 3d path planning based on the improved hierarchical deep q network. J Mar Sci Eng 8(2):145

    Article  Google Scholar 

  24. Zang W, Nie Y, Song D, Guo T, Li K (2019) Research on constraining strategies of underwater glider’s trajectory under the influence of ocean currents based on dqn algorithm. In: OCEANS 2019 MTS/IEEE SEATTLE. IEEE, pp 1–5

  25. Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M (2013) Playing atari with deep reinforcement learning. arXiv:1312.5602

  26. Lv L, Zhang S, Ding D, Wang Y (2019) Path planning via an improved dqn-based learning policy. IEEE Access 7:67319–67330

    Article  Google Scholar 

  27. Moon JH, Jee SC, Lee HJ (2016) Output-feedback control of underwater gliders by buoyancy and pitching moment control: feedback linearization approach. Int J Control Automat Syst 14(1):255–262

    Article  Google Scholar 

  28. Zhang F, Tan X, Khalil HK (2012) Passivity-based controller design for stablization of underwater gliders. In: 2012 American control conference (ACC). IEEE, pp 5408–5413

  29. Zhang S-W, Yu J-C, Zhang A-Q (2012) Optimal control for underwater gliders in the vertical plane. Control Theory Appl 29(1):19–26

    Google Scholar 

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China (Grant No. 51909252), and the Fundamental Research Funds for the Central Universities (Grant No. 202061004). This work is also partly supported by the China Scholarship Council.

Author information

Authors and Affiliations

  1. The College of Information Science and Engineering, Ocean University of China, Songling Road, Qingdao, 266000, Shandong, China

    Wenchuan Zang

  2. The College of Engineering, Ocean University of China, Songling Road, Qingdao, 266000, Shandong, China

    Peng Yao, Kunling Lv & Dalei Song

Authors
  1. Wenchuan Zang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peng Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kunling Lv
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dalei Song
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Wenchuan Zang, software, conceptualization, manuscript preparation-writing. Peng Yao, methodology, conceptualization. Kunling Lv, methodology, manuscript preparation-editing. Dalei Song, manuscript preparation-editing.

Corresponding author

Correspondence to Dalei Song.

Ethics declarations

Conflict of interest

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zang, W., Yao, P., Lv, K. et al. A deep Q network assisted method for underwater gliders standoff tracking to the static target. Neural Comput & Applic 34, 20575–20587 (2022). https://doi.org/10.1007/s00521-022-07408-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07408-w

Keywords

Search

Navigation