Skip to main content
Springer Nature Link
Log in

Real-Time Monocular Obstacle Detection Based on Horizon Line and Saliency Estimation for Unmanned Surface Vehicles

  • Published:
Mobile Networks and Applications Aims and scope Submit manuscript

Abstract

Recently, real-time obstacle detection by monocular vision exhibits a promising prospect in enhancing the safety of unmanned surface vehicles (USVs). Since the obstacles that may threaten USVs generally appear below the water edge, most existing methods first detect the horizon line and then search for obstacles below the estimated horizon line. However, these methods detect horizon line only using edge or line features, which are susceptible to interference edges from clouds, waves, and land, eventually resulting in poor obstacle detection. To avoid being affected by interference edges, in this paper, we propose a novel horizon line detection method based on semantic segmentation. The method assumes a Gaussian mixture model (GMM) with spatial smoothness constraints to fit the semantic structure of marine images and simultaneously generate a water segmentation mask. The horizon line is estimated from the water boundary points via straight line fitting. Further, inspired by human visual attention mechanisms, an efficient saliency detection method based on background prior and contrast prior is presented to detect obstacles below the estimated horizon line. To reduce false positives caused by sun glitter, waves and foam, the continuity of the adjacent frames is employed to filter the detected obstacles. An extensive evaluation was conducted on a large marine image dataset collected by our ‘Jinghai VIII’ USV. The experimental results show that the proposed method significantly outperformed the recent state-of-the-art marine obstacle method by 22.07% in terms of F-score while running over 24 fps on an NVIDIA GTX1080Ti GPU.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Annamalai AS, Sutton R, Yang C, Culverhouse P, Sharma S (2015) Robust adaptive control of an uninhabited surface vehicle. J Intell Robot Syst 78(2):319–338

    Article  Google Scholar 

  2. Liu Y, Bucknall R, Zhang X (2017) The fast marching method based intelligent navigation of an unmanned surface vehicle. Ocean Eng 142:363–376

    Article  Google Scholar 

  3. Han J, Cho Y, Kim J, Kim J, Son NS, Kim SY (2020) Autonomous collision detection and avoidance for ARAGON USV: Development and field tests. Journal of Field Robotics 37(6):987–1002

    Article  Google Scholar 

  4. Liu Z, Zhang Y, Yu X, Yuan C (2016) Unmanned surface vehicles: An overview of developments and challenges. Annu Rev Control 41:71–93

    Article  Google Scholar 

  5. Asvadi A, Garrote L, Premebida C, Peixoto P, Nunes UJ (2018) Multimodal vehicle detection: fusing 3D-LIDAR and color camera data. Pattern Recogn Lett 115:20–29

    Article  Google Scholar 

  6. Gal O (2011) Automatic obstacle detection for USV’s navigation using vision sensors. In: Robotic sailing. Springer, pp 127–140

  7. Wang H, Wei Z, Wang S, Ow CS, Ho KT, Feng B (2011) A vision-based obstacle detection system for unmanned surface vehicle. In: 2011 IEEE 5th International conference on robotics, automation and mechatronics (RAM). IEEE, pp 364–369

  8. Wang H, Wei Z, Ow CS, Ho KT, Feng B, Huang J (2012) Improvement in real-time obstacle detection system for USV. In: 2012 12th International conference on control automation robotics & vision (ICARCV). IEEE, pp 1317–1322

  9. Mou X, Wang H (2016) Image-based maritime obstacle detection using global sparsity potentials. J Inf Commun Converg Eng 14(2):129–135

    Google Scholar 

  10. Bloisi DD, Previtali F, Pennisi A, Nardi D, Fiorini M (2016) Enhancing automatic maritime surveillance systems with visual information. IEEE Trans Intell Transp Syst 18(4):824–833

    Article  Google Scholar 

  11. Zhang Y, Li QZ, Zang FN (2017) Ship detection for visual maritime surveillance from non-stationary platforms. Ocean Eng 141:53–63

    Article  Google Scholar 

  12. Shi J, Jin J, Zhang J (2018) Object detection based on saliency and sea-sky line for USV vision. In: 2018 IEEE 4th Information technology and mechatronics engineering conference (ITOEC). IEEE, pp 1581–1586

  13. Liang D, Zhang W, Huang Q, Yang F (2015) Robust sea-sky-line detection for complex sea background. In: 2015 IEEE International conference on progress in informatics and computing (PIC). IEEE, pp 317–321

  14. Kristan M, Kenk VS, Kovačič S, Perš J (2016) Fast image-based obstacle detection from unmanned surface vehicles. IEEE Trans Cybern 46(3):641–654

    Article  Google Scholar 

  15. Bovcon B, Perš J, Kristan M, et al. (2017) Improving vision-based obstacle detection on USV using inertial sensor. In: Proceedings of the 10th International symposium on image and signal processing and analysis. IEEE, pp 1–6

  16. Bovcon B, Perš J, Kristan M, et al. (2018) Stereo obstacle detection for unmanned surface vehicles by IMU-assisted semantic segmentation. Robot Auton Syst 104:1–13

    Article  Google Scholar 

  17. Liu J, Li H, Luo J, Xie S, Sun Y (2021) Efficient obstacle detection based on prior estimation network and spatially constrained mixture model for unmanned surface vehicles. Journal of Field Robotics 38 (2):212–228

    Article  Google Scholar 

  18. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 779–788

  19. Zhang X, Chen Z, Wu QJ, Cai L, Lu D, Li X (2019) Fast semantic segmentation for scene perception. IEEE Trans Industr Inf 15(2):1183–1192

    Article  Google Scholar 

  20. Chen LC, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision (ECCV), pp 801–818

  21. Choi S, Kim JT, Choo J (2020) Cars can’t fly up in the sky: improving urban-scene segmentation via height-driven attention networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9373–9383

  22. Lee SJ, Roh MI, Lee HW, Ha JS, Woo IG, et al. (2018) Image-based ship detection and classification for unmanned surface vehicle using real-time object detection neural networks. In: Proceedings of the international Offshore and Polar Engineering Conference, pp 726–730

  23. Yang J, Li Y, Zhang Q, Ren Y (2019) Surface vehicle detection and tracking with deep learning and appearance feature. In: 2019 5th International Conference on Control, Automation and Robotics (ICCAR).IEEE, pp 276–280

  24. Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp 91–99

  25. Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Süsstrunk S. (2012) SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans Pattern Anal Mach Intell 34(11):2274–2282

    Article  Google Scholar 

  26. Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395

    Article  MathSciNet  Google Scholar 

  27. Grafarend EW (2006) Linear and nonlinear models: fixed effects, random effects, and mixed models. de Gruyter

  28. Rui Z, Jingyi L, Hengyu L, Qixing C (2020) Real-Time obstacle detection based on monocular vision for unmanned surface vehicles. In: International conference on bio-inspired information and communication technologies. Springer, pp 166–180

  29. Nguyen TM, Wu QJ (2012) Fast and robust spatially constrained Gaussian mixture model for image segmentation. IEEE Trans Circ Syst Video Technol 23(4):621–635

    Article  Google Scholar 

  30. Cheng MM, Mitra NJ, Huang X, Torr PH, Hu SM (2014) Global contrast based salient region detection. IEEE Trans Pattern Anal Mach Intell 37(3):569–582

    Article  Google Scholar 

  31. Borji A, Cheng MM, Hou Q, Jiang H, Li J (2019) Salient object detection: A survey. Comput Vis Media 5(2):117–150

    Article  Google Scholar 

  32. Zhu W, Liang S, Wei Y, Sun J (2014) Saliency optimization from robust background detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2814–2821

  33. Li J, Meng F, Zhang Y (2015) Saliency detection using a background probability model. In: 2015 IEEE international conference on image processing (ICIP). IEEE, pp 2189–2193

  34. Wang J, Lu H, Li X, Tong N, Liu W (2015) Saliency detection via background and foreground seed selection. Neurocomputing 152:359–368

    Article  Google Scholar 

  35. Perazzi F, Krähenbühl P, Pritch Y, Hornung A (2012) Saliency filters: Contrast based filtering for salient region detection. In: 2012 IEEE conference on computer vision and pattern recognition. IEEE, pp 733–740

  36. Cheng MM, Mitra NJ, Huang X, Torr PH, Hu SM (2015) Global contrast based salient region detection. IEEE Trans Pattern Anal Mach Intell 37(3):569–582

    Article  Google Scholar 

  37. Li F, Zhang J, Sun W, Jin J, Li L, Dai Y (2021) Sea–sky line detection using gray variation differences in the time domain for unmanned surface vehicles. SIViP 15(1):139–146

Download references

Acknowledgements

This work was supported by the National Key Research and Development Program of China (No. 2018YFB1304503), the Natural Science Foundation of Shanghai of China (No. 18ZR1415300), and the Key Research and Development Program of Jiangxi Province of China (No. 20192BBEL50004). The authors also gratefully acknowledge the helpful comments and suggestions of the editor and anonymous reviewers, which have improved the presentation.

Author information

Authors and Affiliations

  1. School of Mechatronic Engineering and Automation, Shanghai University, 99 Shangda Road, Shanghai, 200444, China

    Jingyi Liu, Hengyu Li, Jun Liu, Shaorong Xie & Jun Luo

  2. Department of Mathematics, Jining University, Qufu, Shandong Province, 273155, China

    Jun Liu

Authors
  1. Jingyi Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Hengyu Li
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Jun Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Shaorong Xie
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Jun Luo
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Hengyu Li.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, J., Li, H., Liu, J. et al. Real-Time Monocular Obstacle Detection Based on Horizon Line and Saliency Estimation for Unmanned Surface Vehicles. Mobile Netw Appl 26, 1372–1385 (2021). https://doi.org/10.1007/s11036-021-01752-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11036-021-01752-2

Keywords