Skip to main content

Advertisement

SpringerLink
Log in

Image blending-based noise synthesis and attention-guided network for single image marine snow denoising

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

In this paper, we consider the problem of single image marine snow denoising. Due to the complex underwater environment, the structure and statistical properties of marine snow noise are substantially different from those of the noises encountered in the atmosphere. To synthesize realistic noisy-clean image pairs for training, we propose an image blending-based noise synthesis method that can better simulate the marine snow without training deep networks. Specifically, the noise is first cropped from a real noisy image and pasted into a clean image to produce a composite image. Then, the Gaussian–Poisson Equation is employed to generate a well-blended image (synthetic noisy image). Furthermore, we introduce an attention-guided denoising network that leverages the location information of noise. The proposed network can detect the marine snow noise and then remove it guided by the estimated attention map. Extensive experiments on synthetic and real-world datasets demonstrate that our denoising network can effectively remove the marine snow noise, while preserving rich details of backgrounds. Other alternative marine snow synthesis approaches are also compared to show the superiority of our noise synthesis method in terms of visual quality and running time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

The datasets analysed during the current study are available from the corresponding author on reasonable request.

References

  1. Wang Y, Tang C, Cai M, Yin J, Wang S, Cheng L, Wang R, Tan M (2020) Real-time underwater onboard vision sensing system for robotic gripping. IEEE Trans Instrum Meas 70:1–11

    Article  Google Scholar 

  2. Han M, Lyu Z, Qiu T, Xu M (2020) A review on intelligence dehazing and color restoration for underwater images. IEEE Trans Syst Man Cybern Syst 50(5):1820–1832

    Article  Google Scholar 

  3. Jiang Q, Zhang Y, Bao F, Zhao X, Zhang C, Liu P (2022) Two-step domain adaptation for underwater image enhancement. Pattern Recognit 122:108324

    Article  Google Scholar 

  4. Hou G, Li J, Wang G, Pan Z, Zhao X (2020) Underwater image dehazing and denoising via curvature variation regularization. Multimed Tools Appl 79(27):20199–20219

    Article  Google Scholar 

  5. Pipara A, Oza U, Mandal S (2021) Underwater image color correction using ensemble colorization network. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 2011–2020

  6. Hou G, Pan Z, Wang G, Yang H, Duan J (2019) An efficient nonlocal variational method with application to underwater image restoration. Neurocomputing 369:106–121

    Article  Google Scholar 

  7. Schettini R, Corchs S (2010) Underwater image processing: state of the art of restoration and image enhancement methods. EURASIP J Adv Signal Process 2010:1–14

    Article  Google Scholar 

  8. Trudnowska E, Lacour L, Ardyna M, Rogge A, Irisson JO, Waite AM, Babin M, Stemmann L (2021) Marine snow morphology illuminates the evolution of phytoplankton blooms and determines their subsequent vertical export. Nat Commun 12(1):1–13

    Article  Google Scholar 

  9. Zhang H, Patel VM (2018) Density-aware single image de-raining using a multi-stream dense network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 695–704

  10. Yang W, Tan RT, Wang S, Liu J (2020) Self-learning video rain streak removal: when cyclic consistency meets temporal correspondence. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1720–1729

  11. Chen W-T, Fang H-Y, Ding J-J, Tsai C-C, Kuo S-Y (2020) Jstasr: joint size and transparency-aware snow removal algorithm based on modified partial convolution and veiling effect removal. In: Proceedings of the European conference on computer vision, pp 754–770. Springer

  12. Tian J, Han Z, Ren W, Chen X, Tang Y (2018) Snowflake removal for videos via global and local low-rank decomposition. IEEE Trans Multimed 20(10):2659–2669

    Article  Google Scholar 

  13. Tian C, Xu Y, Li Z, Zuo W, Fei L, Liu H (2020) Attention-guided cnn for image denoising. Neural Netw 124:117–129

    Article  Google Scholar 

  14. Ko K, Koh YJ, Kim C-S (2022) Blind and compact denoising network based on noise order learning. IEEE Trans Image Process 31:1657–1670

    Article  Google Scholar 

  15. Liang Z, Wang Y, Ding X, Mi Z, Fu X (2021) Single underwater image enhancement by attenuation map guided color correction and detail preserved dehazing. Neurocomputing 425:160–172

    Article  Google Scholar 

  16. Kumar N, Sardana HK, Shome S, Mittal N (2020) Saliency subtraction inspired automated event detection in underwater environments. Cogn Comput 12(1):115–127

    Article  Google Scholar 

  17. Cyganek B, Gongola K (2018) Real-time marine snow noise removal from underwater video sequences. J Electron Imaging 27(4):043002

    Article  Google Scholar 

  18. Liu H, Chau L-P (2019) Deepsea video descattering. Multimed Tools Appl 78(20):28919–28929

    Article  Google Scholar 

  19. Arredondo M, Lebart K (2005) A methodology for the systematic assessment of underwater video processing algorithms. In: Europe Oceans 2005, vol 1, pp 362–367. IEEE

  20. Fier R, Albu AB, Hoeberechts M (2014) Automatic fish counting system for noisy deep-sea videos. In: 2014 Oceans-St. John’s, pp 1–6. IEEE

  21. Jiang Q, Chen Y, Wang G, Ji T (2020) A novel deep neural network for noise removal from underwater image. Signal Process. Image Commun. 87:115921

    Article  Google Scholar 

  22. Wang Y, Yu X, An D, Wei Y (2021) Underwater image enhancement and marine snow removal for fishery based on integrated dual-channel neural network. Comput. Electron. Agric. 186:106182

    Article  Google Scholar 

  23. Chen Y, Sun J, Jiao W, Zhong G (2019) Recovering super-resolution generative adversarial network for underwater images. In: International conference on neural information processing, pp 75–83. Springer

  24. Boffety M, Galland F (2012) Phenomenological marine snow model for optical underwater image simulation: applications to color restoration. In: 2012 Oceans-Yeosu, pp 1–6. IEEE

  25. Boffety M, Galland F, Allais A-G (2012) Color image simulation for underwater optics. Appl Opt 51(23):5633–5642

    Article  Google Scholar 

  26. Li R, Cheong L-F, Tan RT (2019) Heavy rain image restoration: integrating physics model and conditional adversarial learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1633–1642

  27. Wu H, Zheng S, Zhang J, Huang K (2019) Gp-gan: towards realistic high-resolution image blending. In: Proceedings of the 27th ACM international conference on multimedia, pp 2487–2495

  28. Cong W, Zhang J, Niu L, Liu L, Ling Z, Li W, Zhang L (2020) Dovenet: deep image harmonization via domain verification. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8394–8403

  29. Ling J, Xue H, Song L, Xie R, Gu X (2021) Region-aware adaptive instance normalization for image harmonization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9361–9370

  30. Guo Z, Guo D, Zheng H, Gu Z, Zheng B, Dong J (2021) Image harmonization with transformer. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 14870–14879

  31. Chen X, Xu C, Yang X, Tao D (2018) Attention-gan for object transfiguration in wild images. In: Proceedings of the European conference on computer vision, pp 164–180

  32. Jiang K, Wang Z, Yi P, Chen C, Huang B, Luo Y, Ma J, Jiang J (2020) Multi-scale progressive fusion network for single image deraining. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8346–8355

  33. Wang Z, Cun X, Bao J, Zhou W, Liu J, Li H (2022) Uformer: a general u-shaped transformer for image restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 17683–17693

  34. Chen L, Chu X, Zhang X, Sun J (2022) Simple baselines for image restoration. In: Proceedings of the European conference on computer vision

  35. Slade WH, Boss E, Russo C (2011) Effects of particle aggregation and disaggregation on their inherent optical properties. Opt. Express 19(9):7945–7959

    Article  Google Scholar 

  36. Hodne LM, Leikvoll E, Yip M, Teigen AL, Stahl A, Mester R (2022) Detecting and suppressing marine snow for underwater visual slam. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5101–5109

  37. Zhu J-Y, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision, pp 2223–2232

  38. Jeong S, Kim Y, Lee E, Sohn K (2021) Memory-guided unsupervised image-to-image translation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6558–6567

  39. Han J, Shoeiby M, Petersson L, Armin MA (2021) Dual contrastive learning for unsupervised image-to-image translation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 746–755

  40. Chen R, Huang W, Huang B, Sun F, Fang B (2020) Reusing discriminators for encoding: towards unsupervised image-to-image translation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8168–8177

  41. Upchurch P, Gardner J, Pleiss G, Pless R, Snavely N, Bala K, Weinberger K (2017) Deep feature interpolation for image content changes. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7064–7073

  42. Arjovsky M, Chintala S, Bottou L (2017) Wasserstein generative adversarial networks. In: International conference on machine learning, pp 214–223. PMLR

  43. Mäkinen Y, Azzari L, Foi A (2020) Collaborative filtering of correlated noise: exact transform-domain variance for improved shrinkage and patch matching. IEEE Trans Image Process 29:8339–8354

    Article  MATH  Google Scholar 

  44. Zhang K, Zuo W, Chen Y, Meng D, Zhang L (2017) Beyond a gaussian denoiser: residual learning of deep cnn for image denoising. IEEE Trans Image Process 26(7):3142–3155

    Article  MathSciNet  MATH  Google Scholar 

  45. You C, Han L, Feng A, Zhao R, Tang H, Fan W (2022) Megan: memory enhanced graph attention network for space-time video super-resolution. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 1401–1411

  46. Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1357–1366

  47. Ren D, Zuo W, Hu Q, Zhu P, Meng D (2019) Progressive image deraining networks: a better and simpler baseline. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3937–3946

  48. Ye Y, Chang Y, Zhou H, Yan L (2021) Closing the loop: joint rain generation and removal via disentangled image translation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2053–2062

  49. Pérez P, Gangnet M, Blake A (2003) Poisson image editing. In: ACM SIGGRAPH 2003 Papers, pp 313–318

  50. Frankot RT, Chellappa R (1988) A method for enforcing integrability in shape from shading algorithms. IEEE Trans Pattern Anal Mach Intell 10(4):439–451

    Article  MATH  Google Scholar 

  51. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  52. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  53. Xingjian S, Chen Z, Wang H, Yeung D-Y, Wong W-K, Woo W-c (2015) Convolutional lstm network: a machine learning approach for precipitation nowcasting. In: Advances in neural information processing systems, pp 802–810

  54. Liu C, Li H, Wang S, Zhu M, Wang D, Fan X, Wang Z (2021) A dataset and benchmark of underwater object detection for robot picking. In: 2021 IEEE international conference on multimedia and expo workshops, pp 1–6. IEEE

  55. Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp 248–255. IEEE

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant No. 41876098) and the Shenzhen Science and Technology Project (Grant No. JCYJ20200109143041798).

Author information

Authors and Affiliations

  1. Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, 518055, China

    Zeyu Zhao & Xiu Li

  2. Department of Automation, Tsinghua University, Beijing, 100084, China

    Zeyu Zhao

Authors
  1. Zeyu Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiu Li.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, Z., Li, X. Image blending-based noise synthesis and attention-guided network for single image marine snow denoising. Int. J. Mach. Learn. & Cyber. 14, 2205–2219 (2023). https://doi.org/10.1007/s13042-022-01756-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-022-01756-8

Keywords

Search

Navigation