Skip to main content

Advertisement

SpringerLink
Log in

Saliency Subtraction Inspired Automated Event Detection in Underwater Environments

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Unmanned underwater exploration in unconstrained environments is a challenging problem. Analysis of the large volumes of images/videos captured by underwater stations/vehicles manually is a major bottleneck for further research. Existing computer vision methods either do not target unconstrained underwater environments or they only aim to detect static or moving entities. In this paper, we present a novel method for analyzing underwater videos and detecting events. Entry/exit of an object in scene is treated as an event independent of the other objects present therein. The method is applied on underwater videos with no prior knowledge, thus aiding in automated underwater exploration. The method is inspired by the fact that saliency of objects in the scene is invariant of the surrounding environment. The proposed method is composed of three main steps: Local Patch Saliency, Adaptive Saliency Subtraction, and event generation for analyzing underwater imagery from the videos. The method is aimed at detecting overlapping events containing man-made as well as natural objects including those containing multiple objects in the unconstrained underwater conditions. The performance of the method is evaluated on publicly available videos obtained from Ocean Networks Canada and Fish4Knowledge datasets. Ground truth for Ocean Networks Canada videos is not available; hence, a method for generating the same for varied sources is also presented. The algorithm achieves a precision of 98% for event detection with 20% misclassification rate. The results show the robustness of the method that performs even in complex and varying underwater conditions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Yuh J. Design and control of autonomous underwater robots: a survey. Auton Robots 2000;8(1):7–24.

    Article  Google Scholar 

  2. Griffiths G (ed.) Technology and applications of autonomous underwater vehicles. CRC Press; 2002 Nov 28.

  3. Tu Z, Abel A, Zhang L, Luo B, Hussain A. A new spatio-temporal saliency-based video object segmentation. Cogn Comput 2016;8(4):629–47.

    Article  Google Scholar 

  4. Aboudib A, Gripon V, Coppin G. A biologically inspired framework for visual information processing and an application on modeling bottom-up visual attention. Cogn Comput 2016;8(6):1007–26.

    Article  Google Scholar 

  5. Jia X, Li X, Jin Y, Miao J. Region-enhanced multi-layer extreme learning machine. Cogn Comput 2019;11(1):101–9.

    Article  Google Scholar 

  6. Wang H, Xu L, Wang X, Luo B. Learning optimal seeds for ranking saliency. Cogn Comput 2018; 10(2):347–58.

    Article  Google Scholar 

  7. Zheng A, Xu M, Luo B, Zhou Z, Li C. CLASS collaborative low-rank and sparse separation for moving object detection. Cognit Comput 2017;9(2):180–93.

    Article  Google Scholar 

  8. Olmos A, Trucco E. 2002. Detecting man-made objects in unconstrained subsea videos. In: BMVC. p. 1–10.

  9. Edgington DR, Salamy KA, Risi M, Sherlock RE, Walther D, Koch C. Automated event detection in underwater video. Oceans 2003. Celebrating the past teaming toward the future (IEEE Cat. No. 03CH37492). IEEE; 2003. p. P2749–53.

  10. Walther D, Edgington DR, Koch C. Detection and tracking of objects in underwater video. Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, 2004. CVPR 2004. IEEE; 2004. p. I–I.

  11. Kabatek M, Azimi-Sadjadi MR, Tucker JD. An underwater target detection system for electro-optical imagery data. OCEANS. IEEE; 2009. p. 1–8.

  12. Cucchiara R, Grana C, Piccardi M, Prati A. Detecting moving objects, ghosts, and shadows in video streams. IEEE Trans Pattern Anal Mach Intell 2003;25(10):1337–42.

    Article  Google Scholar 

  13. Lipton AJ, Fujiyoshi H, Patil RS. Moving target classification and tracking from real-time video. Proceedings fourth IEEE workshop on applications of computer vision. WACV’98 (Cat. No. 98EX201). IEEE; 1998. p. 8–14.

  14. Piccardi M. Background subtraction techniques: a review. 2004 IEEE International conference on systems, man and cybernetics (IEEE Cat. No. 04CH37583). IEEE; 2004. p. 3099–104.

  15. Spampinato C, Palazzo S. Enhancing object detection performance by integrating motion objectness and perceptual organization. Proceedings of the 21st international conference on pattern recognition (ICPR2012). IEEE; 2012. p. 3640–3.

  16. Palazzo S, Kavasidis I, Spampinato C. Covariance based modeling of underwater scenes for fish detection. 2013 IEEE International conference on image processing. IEEE; 2013. p. 1481–5.

  17. Spampinato C, Palazzo S, Kavasidis I. A texton-based kernel density estimation approach for background modeling under extreme conditions. Comput Vis Image Understand 2014;122:74–83.

    Article  Google Scholar 

  18. Heikkilä M, Pietikäinen M, Schmid C. Description of interest regions with local binary patterns. Pattern Recogn 2009;42(3):425–36.

    Article  Google Scholar 

  19. Liao S, Zhao G, Kellokumpu V, Pietikäinen M, Li SZ. Modeling pixel process with scale invariant local patterns for background subtraction in complex scenes. 2010 IEEE Computer society conference on computer vision and pattern recognition. IEEE; 2010. p. 1301–6.

  20. Spampinato C, Chen-Burger YH, Nadarajan G, Fisher RB. Detecting, tracking and counting fish in low quality unconstrained underwater videos. VISAPP (2) 2008;2008(514–9):1.

    Google Scholar 

  21. Spampinato C, Beauxis-Aussalet E, Palazzo S, Beyan C, van Ossenbruggen J, He J, Boom B, Huang X. A rule-based event detection system for real-life underwater domain. Mach Vis Appl 2014;25(1): 99–117.

    Article  Google Scholar 

  22. Akilan T, Wu QJ, Yang Y. Fusion-based foreground enhancement for background subtraction using multivariate multi-model Gaussian distribution. Inform Sci 2018;430:414–31.

    Article  Google Scholar 

  23. Minematsu T, Shimada A, Uchiyama H, Taniguchi RI. Analytics of deep neural network-based background subtraction. J Imag 2018;4(6):78.

    Article  Google Scholar 

  24. Rova A, Mori G, Dill LM. One fish, two fish, butterfish, trumpeter: recognizing fish in underwater video. InMVA. 2007; 404–7.

  25. Shortis MR, Ravanbakhsh M, Shafait F, Mian A. Progress in the automated identification, measurement, and counting of fish in underwater image sequences. Marine Technol Soc J 2016;50(1):4–16.

    Article  Google Scholar 

  26. Vasamsetti S, Mittal N, Neelapu BC, Sardana HK. 3D local spatio-temporal ternary patterns for moving object detection in complex scenes. Cogn Comput 2019;11(1):18–30.

    Article  Google Scholar 

  27. Oliver K, Hou W, Wang S. 2010. Image feature detection and matching in underwater conditions, Vol. 7678: International Society for Optics and Photonics.

  28. Kavasidis I, Palazzo S. Quantitative performance analysis of object detection algorithms on underwater video footage. Proceedings of the 1st ACM international workshop on multimedia analysis for ecological data. ACM; 2012. p. 57–60.

  29. Han KM, Choi HT. Shape context based object recognition and tracking in structured underwater environment. 2011 IEEE International geoscience and remote sensing symposium. IEEE; 2011. p. 617–20.

  30. Kim D, Lee D, Myung H, Choi HT. Object detection and tracking for autonomous underwater robots using weighted template matching. 2012 Oceans-Yeosu. IEEE; 2012. p. 1–5.

  31. Leonard I, Arnold-Bos A, Alfalou A. Interest of correlation-based automatic target recognition in underwater optical images: theoretical justification and first results. Ocean sensing and monitoring II. International Society for Optics and Photonics; 2010. p. 76780O.

  32. Barat C, Rendas MJ. A robust visual attention system for detecting manufactured objects in underwater video. OCEANS 2006. IEEE; 2006. p. 1–6.

  33. Barat C, Phlypo R. A fully automated method to detect and segment a manufactured object in an underwater color image. EURASIP J Adv Signal Process 2010;1:10.

    Google Scholar 

  34. Itti L, Koch C, Niebur E. A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell 1998;1(11):1254–9.

    Article  Google Scholar 

  35. Wang HB, Dong X, Shen J, Wu XW, Chen Z. Saliency-based adaptive object extraction for color underwater images. Applied mechanics and materials. Trans Tech Publications; 2013. p. 3964–70.

  36. Bazeille S, Quidu I, Jaulin L. Identification of underwater man-made object using a colour criterion. Conference on detection and classification of underwater targets; 2007. p. xx.

  37. Maldonado-Ramírez A, Torres-Méndez LA. 2016. Robotic visual tracking of relevant cues in underwater environments with poor visibility conditions. J Sensors.

  38. Gebali A, Albu AB, Hoeberechts M. Detection of salient events in large datasets of underwater video. IEEE; 2012. p. 14.

  39. Oliva A, Torralba A, Castelhano MS, Henderson JM. Top-down control of visual attention in object detection. Proceedings 2003 international conference on image processing (Cat. No. 03CH37429). IEEE; 2003. p. I-253.

  40. Rutishauser U, Walther D, Koch C, Perona P. Is bottom-up attention useful for object recognition? Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, 2004. CVPR 2004. IEEE; 2004. p. II-II.

  41. Bosch A, Zisserman A, Muñoz X. Scene classification using a hybrid generative/discriminative approach. IEEE Trans Pattern Anal Mach Intell 2008;30(4):712–27.

    Article  Google Scholar 

  42. Kinchla RA, Wolfe JM. The order of visual processing:”Top-down,””bottom-up,” or ”middle-out”. Percept Psychophys 1979;25(3):225–31.

    Article  CAS  Google Scholar 

  43. Zelnik-Manor L, Irani M. Event-based analysis of video. InCVPR (2) 2001;8:123–30.

    Google Scholar 

  44. Ke Y, Sukthankar R, Hebert M. Event detection in crowded videos. 2007 IEEE 11th international conference on computer vision. IEEE; 2007. p. 1–8.

  45. Borji A, Itti L. State-of-the-art in visual attention modeling. IEEE Trans Pattern Anal Mach Intell 2013; 35(1):185–207.

    Article  Google Scholar 

  46. Li H, Ngan KN. A co-saliency model of image pairs. IEEE Trans Image Process 2011;20(12):3365–75.

    Article  Google Scholar 

  47. Meng F, Li H, Liu G, Ngan KN. Object co-segmentation based on shortest path algorithm and saliency model. IEEE Trans Multimed 2012;14(5):1429–41.

    Article  Google Scholar 

  48. Borji A, Itti L. Exploiting local and global patch rarities for saliency detection. 2012 IEEE conference on computer vision and pattern recognition. IEEE; 2012. p. 478–85.

  49. Borji A, Sihite DN, Itti L. Quantitative analysis of human-model agreement in visual saliency modeling: a comparative study. IEEE Trans Image Process 2013;22(1):55–69.

    Article  Google Scholar 

  50. Lin Y, Tong Y, Cao Y, Zhou Y, Wang S. Visual-attention-based background modeling for detecting infrequently moving objects. IEEE Trans Circ Syst Video Technol 2017;27(6):1208–21.

    Article  Google Scholar 

  51. Sezgin M, Sankur B. Survey over image thresholding techniques and quantitative performance evaluation. J Electron Imag 2004;13(1):146–66.

    Article  Google Scholar 

  52. Kumar N, Sardana HK, Shome SN. Saliency based shape extraction of objects in unconstrained underwater environment. Multimed Tools Appl 2019;78(11):15121–39.

    Article  Google Scholar 

  53. Kavasidis I, Palazzo S, Di Salvo R, Giordano D, Spampinato C. An innovative web-based collaborative platform for video annotation. Multimed Tools Appl 2014;70(1):413–32.

    Article  Google Scholar 

  54. Barnes CR, Best MM, Bornhold BD, Juniper SK, Pirenne B, Phibbs P. The NEPTUNE project-a cabled ocean observatory in the NE Pacific: overview, challenges and scientific objectives for the installation and operation of Stage I in Canadian waters. 2007 Symposium on underwater technology and workshop on scientific use of submarine cables and related technologies. IEEE; 2007. p. 308–13.

Download references

Acknowledgments

The authors are grateful to Dr. Maia Hoeberechts and team for providing the Ocean Networks Canada Dataset. We are also thankful to Akanksha Pathania, Parminder Kaur, Gifty Aggarwal, and Neha for assisting us in generating the ground truth for the underwater videos.

Funding

Nitin Kumar is thankful to the Council of Scientific and Industrial Research - Central Scientific Instruments Organisation (CSIR-CSIO), Chandigarh for providing the funding and opportunity to carry out this work at CSIR-CSIO.

Author information

Authors and Affiliations

  1. Academy of Scientific and Innovative Research (AcSIR), New Delhi, India

    Nitin Kumar, H. K. Sardana & S. N. Shome

  2. CSIR-Central Scientific Instruments Organisation (CSIR-CSIO), Chandigarh, India

    H. K. Sardana & Neerja Mittal

  3. CSIR-Central Mechanical Engineering Research Institute (CSIR-CMERI), Durgapur, West Bengal, India

    S. N. Shome

Authors
  1. Nitin Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. H. K. Sardana
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. S. N. Shome
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Neerja Mittal
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to H. K. Sardana.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed Consent

Informed consent is not necessary for the present study.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kumar, N., Sardana, H.K., Shome, S.N. et al. Saliency Subtraction Inspired Automated Event Detection in Underwater Environments. Cogn Comput 12, 115–127 (2020). https://doi.org/10.1007/s12559-019-09671-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-019-09671-x

Keywords

Search

Navigation