Skip to main content
SpringerLink
Log in

Estimating Obstacle Maps for USVs Based on a Multistage Feature Aggregation and Semantic Feature Separation Network

  • Regular Paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Obstacle map estimation based on efficient semantic segmentation networks is promising for improving the environmental awareness of unmanned surface vehicles (USVs). However, existing networks perform poorly in challenging scenes with small obstacles, scenery reflections, boat wakes, and visual ambiguities caused by unfavorable weather conditions. In this paper, we address the small obstacle segmentation problem by learning representations of obstacles at multiple scales. An efficient multistage feature aggregation (MFA) module is proposed, which utilizes fully separable convolutions of different sizes to capture and fuse multiscale context information from different stages of a backbone network. In addition, a novel feature separation (FS) loss function based on Gaussian mixture model is presented, which encourages the MFA module to enforce separation among different semantic features, thereby providing a robust and discriminative representation in various challenging scenes. Building upon the MFA module and the FS loss function, we present a fast multistage feature aggregation and semantic feature separation network (FASNet) for obstacle map estimation of USVs. An extensive evaluation was conducted on a challenging public dataset (MaSTr1325). We validated that various lightweight semantic segmentation models achieved consistent performance improvement when our MFA module and FS loss function were adopted. The evaluation results showed that the proposed FASNet outperformed state-of-the-art lightweight models and achieved 96.71% mIoU and > 1.5% higher obstacle-class IoU than the second-best network, while running over 58 fps.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Availability of data and material

The MaSTr1325 dataset that was used to train and evaluate FASNet is made publicly available at https://vicos.si/Projects/Viamaro.

Code Availability

A public version of FASNet is available at https://github.com/aluckyi/FASNet.

References

  1. Kristan, M., Kenk, V.S., Kovačič, S., Perš, J.: Fast image-based obstacle detection from unmanned surface vehicles. IEEE Trans. Cybern. 46(3), 641–654 (2016)

    Article  Google Scholar 

  2. Bovcon, B., Perš, J., Kristan, M., et al.: Improving vision-based obstacle detection on USV using inertial sensor. In: Image and Signal Processing and Analysis (ISPA), 2017 10th International Symposium on, pp. 1–6. IEEE (2017)

  3. Bovcon, B., Mandeljc, R., Perš, J., Kristan, M.: Stereo obstacle detection for unmanned surface vehicles by IMU-assisted semantic segmentation. Robot. Auton. Syst. 104, 1–13 (2018)

    Article  Google Scholar 

  4. Bovcon, B., Kristan, M.: Obstacle detection for USVs by joint stereo-view semantic segmentation. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5807–5812. IEEE (2018)

  5. Liu, J., Li, H., Luo, J., Xie, S., Sun, Y.: Efficient obstacle detection based on prior estimation network and spatially constrained mixture model for unmanned surface vehicles. J. Field Robot. 38(2), 212–228 (2021)

    Article  Google Scholar 

  6. Zhao, Y., Li, W., Shi, P.: A real-time collision avoidance learning system for Unmanned Surface Vessels. Neurocomputing 182, 255–266 (2016)

    Article  Google Scholar 

  7. Chen, J., Pan, W., Guo, Y., Huang, C., Wu, H.: An obstacle avoidance algorithm designed for USV based on single beam sonar and fuzzy control. In: 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 2446–2451. IEEE (2013)

  8. Onunka, C., Bright, G.: Autonomous marine craft navigation: On the study of radar obstacle detection. In: 2010 11th International Conference on Control Automation Robotics & Vision, pp. 567–572. IEEE (2010)

  9. Peng, Y., Yang, Y., Cui, J., Li, X., Pu, H., Gu, J., Xie, S., Luo, J.: Development of the USV ‘JingHai-I’ and sea trials in the Southern Yellow Sea. Ocean engineering 131, 186–196 (2017)

    Article  Google Scholar 

  10. Bloisi, D.D., Previtali, F., Pennisi, A., Nardi, D., Fiorini, M.: Enhancing automatic maritime surveillance systems with visual information. IEEE Trans. Intell. Transp. Syst. 18(4), 824–833 (2016)

    Article  Google Scholar 

  11. Zhang, Y., Li, Q.Z., Zang, F.N.: Ship detection for visual maritime surveillance from non-stationary platforms. Ocean Eng. 141, 53–63 (2017)

    Article  Google Scholar 

  12. Mou, X., Wang, H.: Image-based maritime obstacle detection using global sparsity potentials. J. Inf. Commun. Converg. Eng. 14(2), 129–135 (2016)

    Google Scholar 

  13. Wang, H., Wei, Z., Wang, S., Ow, C.S., Ho, K.T., Feng, B., Lubing, Z.: Real-time obstacle detection for unmanned surface vehicle. In: Defense Science Research Conference and Expo (DSR), vol. 2011, pp. 1–4. IEEE (2011)

  14. Shi, J., Jin, J., Zhang, J.: Object detection based on saliency and sea-sky line for USV vision. In: 2018 IEEE 4th Information Technology and Mechatronics Engineering Conference (ITOEC), pp. 1581–1586. IEEE (2018)

  15. Feng, D., Haase-Schütz, C., Rosenbaum, L., Hertlein, H., Glaeser, C., Timm, F., Wiesbeck, W., Dietmayer, K.: Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges. IEEE Trans. Intell. Transp. Syst. (2020)

  16. Paszke, A., Chaurasia, A., Kim, S., Culurciello, E.: Enet: A deep neural network architecture for real-time semantic segmentation. arXiv:1606.02147 (2016)

  17. Zhang, X., Chen, Z., Wu, Q.J., Cai, L., Lu, D., Li, X.: Fast Semantic Segmentation for Scene Perception. IEEE Trans. Ind. Inf. 15(2), 1183–1192 (2019)

    Article  Google Scholar 

  18. Romera, E., Alvarez, J.M., Bergasa, L.M., Arroyo, R.: Erfnet: Efficient residual factorized convnet for real-time semantic segmentation. IEEE Trans. Intell. Transp. Syst. 19(1), 263–272 (2017)

    Article  Google Scholar 

  19. Wang, Y., Zhou, Q., Xiong, J., Wu, X., Jin, X.: ESNet: An efficient symmetric network for real-time semantic segmentation. In: Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pp. 41–52. Springer (2019)

  20. Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2881–2890 (2017)

  21. Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 801–818 (2018)

  22. Li, X., Zhong, Z., Wu, J., Yang, Y., Lin, Z., Liu, H.: Expectation-maximization attention networks for semantic segmentation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 9167–9176 (2019)

  23. Bovcon, B., Muhovič, J., Perš, J., Kristan, M.: The MaSTr1325 dataset for training deep USV obstacle detection models. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3431–3438. IEEE (2019)

  24. Bovcon, B., Kristan, M.: Benchmarking semantic segmentation methods for obstacle detection on a marine environment. In: 24th Computer Vision Winter Workshop, pp. 1–9 (2019)

  25. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234–241. Springer (2015)

  26. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 834–848 (2017)

    Article  Google Scholar 

  27. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3213–3223 (2016)

  28. Brostow, G.J., Shotton, J., Fauqueur, J., Cipolla, R.: Segmentation and recognition using structure from motion point clouds. In: European Conference on Computer Vision, pp. 44–57. Springer (2008)

  29. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  30. Chollet, F.: Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1251–1258 (2017)

  31. Cane, T., Ferryman, J.: Evaluating deep semantic segmentation networks for object detection in maritime surveillance. In: 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1–6. IEEE (2018)

  32. Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2117–2125 (2017)

  33. Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167 (2015)

  34. He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)

  35. Fan, X., Yang, Y., Deng, C., Xu, J., Gao, X.: Compressed multi-scale feature fusion network for single image super-resolution. Signal Process. 146, 50–60 (2018)

    Article  Google Scholar 

  36. Yu, C., Wang, J., Gao, C., Yu, G., Shen, C., Sang, N.: Context Prior for Scene Segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12416–12425 (2020)

  37. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)

  38. Nguyen, T.M., Wu, Q.J.: Fast and robust spatially constrained Gaussian mixture model for image segmentation. IEEE Trans. Circuits Syst. Video Technol. 23(4), 621–635 (2013)

    Article  Google Scholar 

  39. Garcia-Garcia, A., Orts-Escolano, S., Oprea, S., Villena-Martinez, V., Martinez-Gonzalez, P., Garcia-Rodriguez, J.: A survey on deep learning techniques for image and video semantic segmentation. Appl. Soft Comput. 70, 41–65 (2018)

    Article  Google Scholar 

  40. Fritsch, J., Kuehnl, T., Geiger, A.: A new performance measure and evaluation benchmark for road detection algorithms. In: 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013), pp. 1693–1700. IEEE (2013)

Download references

Acknowledgements

The authors thank Xiaokang Yang and Rui Zhang for their assistance in the experimental evaluation. The authors also gratefully acknowledge the helpful comments and suggestions of the editor and anonymous reviewers, which have improved the presentation.

Funding

This work was supported in part by the National Key Research and Development Program of China (grant number 2018YFB1304503), the National Natural Science Foundation of China (grant numbers 6193308 and 61625304) and the Shanghai Natural Science Foundation (grant number 18ZR1415300).

Author information

Authors and Affiliations

  1. School of Mechatronic Engineering and Automation, Shanghai University, 99 Shangda Road, Shanghai, 200444, China

    Jingyi Liu, Hengyu Li, Jun Luo & Shaorong Xie

  2. Department of Mechanical and Industrial Engineering, University of Toronto, Toronto, Ontario, Canada

    Yu Sun

Authors
  1. Jingyi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hengyu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jun Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shaorong Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yu Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

J. L. (Jingyi Liu) and H. L. conceived the idea. J. L. (Jingyi Liu), H. L. and J. L. (Jun Luo) designed the experiments. J. L. (Jingyi Liu) carried out programming, adjustment and data analysis. J. L. (Jingyi Liu) and Y. S. wrote the manuscript. All authors reviewed the final manuscript.

Corresponding author

Correspondence to Hengyu Li.

Ethics declarations

Conflict of Interests

The authors declare no conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, J., Li, H., Luo, J. et al. Estimating Obstacle Maps for USVs Based on a Multistage Feature Aggregation and Semantic Feature Separation Network. J Intell Robot Syst 102, 21 (2021). https://doi.org/10.1007/s10846-021-01395-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10846-021-01395-1

Keywords

Search

Navigation