Skip to main content
SpringerLink
Log in

Underwater image enhancement using scale-patch synergy transformer

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

In the past few years, underwater image enhancement has attracted an increasing amount of research work because it plays an important role in computer vision related underwater tasks, such as aquatic robotics and marine engineering. However, wavelength-dependent light absorption and scattering introduces unpleasant color distortion and reduces the visibility of images in underwater scenes. In this paper, we propose a two-branch multi-scale (MSN) and multi-patch (MPN) synergy network, called Multi-SPNet, which aims to improve the contrast, brightness, and eliminate color distortion of non-uniform degraded underwater images. Specifically, the features extracted from multi-scale and multi-patch branches are interweaved for progressive image enhancement, where the upper and the lower branches utilize efficient Transformer blocks for learning multi-scale representation from low-to-high resolution and aggregating features via multiple image patches from fine to coarse level, respectively. The complementary branches can construct a synergistic merge to employ their mutual benefits for local and non-local pixel interactions. Extensive experiments on synthetic and real-world underwater image datasets clearly prove the effectiveness and superiority of the proposed Multi-SPNet against the state-of-the-art models both qualitatively and quantitatively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Availability of data and materials

The raw data can be shared if the researchers need to do research on relevant topic.

References

  1. Jaffe, J.S.: Underwater optical imaging: the past, the present, and the prospects. IEEE J. Ocean. Eng. 40(3), 683–700 (2014)

    Article  Google Scholar 

  2. Sheinin, M., Schechner, Y.Y.: The next best underwater view. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 3764–3773 (2016)

  3. Lin, W.H., Zhong, J.X., Liu, S., Li, T., Li, G.: Roimix: proposal-fusion among multiple images for underwater object detection. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2588–2592 (2020)

  4. Jesus, A., Zito, C., Tortorici, C., Roura, E., De Masi, G.: Underwater object classification and detection: first results and open challenges. In: OCEANS, pp. 1–6 (2022)

  5. Akkaynak, D., Treibitz, T.: A revised underwater image formation model. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 6723–6732 (2018)

  6. Fu, X., Fan, Z., Ling, M., Huang, Y., Ding, X.: Two-step approach for single underwater image enhancement. In: IEEE international symposium on intelligent signal processing and communication systems (ISPACS), pp. 789–794 (2017)

  7. Fu, X., Cao, X.: Underwater image enhancement with global-local networks and compressed-histogram equalization. Signal Process. Image Commun. 86, 115892 (2020)

    Article  Google Scholar 

  8. Jiang, Q., Zhang, Y., Bao, F., Zhao, X., Zhang, C., Liu, P.: Two-step domain adaptation for underwater image enhancement. Pattern Recogn. 122, 108324 (2022)

    Article  Google Scholar 

  9. Yin, S., Hu, S., Wang, Y., Wang, W., Li, C., Yang, Y.H.: Degradation-aware and color-corrected network for underwater image enhancement. Knowl.-Based Syst. 258, 109997 (2022)

    Article  Google Scholar 

  10. Liang, P., Ding, W., Fan, L., Wang, H., Li, Z., Yang, F., Wang, B., Li, C.: Multi-scale and multi-patch transformer for sandstorm image enhancement. J. Vis. Commun. Image Represent. 89, 103662 (2022)

    Article  Google Scholar 

  11. Li, C.Y., Guo, J.C., Cong, R.M., Pang, Y.W., Wang, B.: Underwater image enhancement by dehazing with minimum information loss and histogram distribution prior. IEEE Trans. Image Process. 25(12), 5664–5677 (2016)

    Article  MathSciNet  Google Scholar 

  12. Song, W., Wang, Y., Huang, D., Tjondronegoro, D.: A rapid scene depth estimation model based on underwater light attenuation prior for underwater image restoration. In: Pacific Rim Conference on Multimedia (PCM), pp. 678–688 (2018)

  13. Drews, P.L., Nascimento, E.R., Botelho, S.S., Campos, M.F.M.: Underwater depth estimation and image restoration based on single images. IEEE Comput. Gr. Appl. 36(2), 24–35 (2016)

    Article  Google Scholar 

  14. Li, J., Skinner, K.A., Eustice, R.M., Johnson-Roberson, M.: 2017 WaterGAN: unsupervised generative network to enable real-time color correction of monocular underwater images. IEEE Robot. Autom. Lett. 3(1), 387–394 (2017)

    Google Scholar 

  15. Li, C., Anwar, S., Porikli, F.: Underwater scene prior inspired deep underwater image and video enhancement. Pattern Recogn. 98, 107038 (2020)

    Article  Google Scholar 

  16. Li, C., Guo, C., Ren, W., Cong, R., Hou, J., Kwong, S., Tao, D.: An underwater image enhancement benchmark dataset and beyond. IEEE Trans. Image Process. 29, 4376–4389 (2019)

    Article  Google Scholar 

  17. Li, C., Anwar, S., Hou, J., Cong, R., Guo, C., Ren, W.: Underwater image enhancement via medium transmission-guided multi-color space embedding. IEEE Trans. Image Process. 30, 4985–5000 (2021)

    Article  Google Scholar 

  18. Shen, Z., Xu, H., Luo, T., Song, Y., He, Z.: UDAformer: underwater image enhancement based on dual attention transformer. Comput. Gr. 111, 77–88 (2023)

    Article  Google Scholar 

  19. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp. 10012–10022 (2021)

  20. Wang, Z., Cun, X., Bao, J., Zhou, W., Liu, J., Li, H.: Uformer: a general u-shaped transformer for image restoration. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 17683–17693 (2022)

  21. Zamir, S. W., Arora, A., Khan, S., Hayat, M., Khan, F. S., Yang, M. H.: Restormer: Efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 5728–5739 (2022)

  22. Peng, L., Zhu, C., Bian, L.: U-shape transformer for underwater image enhancement. IEEE Trans. Image Process. 32, 3066–3079 (2023)

    Article  Google Scholar 

  23. Marques, T. P., Albu, A. B.: L2uwe: a framework for the efficient enhancement of low-light underwater images using local contrast and multi-scale fusion. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 538–539 (2020)

  24. Islam, M.J., Xia, Y., Sattar, J.: Fast underwater image enhancement for improved visual perception. IEEE Robot. Autom. Lett. 5(2), 3227–3234 (2020)

    Article  Google Scholar 

  25. Xiao, Z., Han, Y., Rahardja, S., Ma, Y.: USLN: A statistically guided lightweight network for underwater image enhancement via dual-statistic white balance and multi-color space stretch. arXiv preprint at arXiv:2209.02221. (2022)

  26. Yan, K., Liang, L., Zheng, Z., Wang, G., Yang, Y.: Medium transmission map matters for learning to restore real-world underwater images. Appl. Sci. 12(11), 5420 (2022)

    Article  Google Scholar 

Download references

Funding

This work was supported by the Natural Science Foundation of Ningxia Province (No. 2023AAC03023).

Author information

Authors and Affiliations

  1. School of Electronic and Electrical Engineering, Ningxia University, Yinchuan, 750021, People’s Republic of China

    Lu Fan & Bo Wang

Authors
  1. Lu Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bo Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

L.F. contributed to the study conception, design and wrote original draft. B.W. completed review and editing of abstract, introduction, related works, methods, experiments, conclusion and references. All authors reviewed the manuscript.

Corresponding author

Correspondence to Bo Wang.

Ethics declarations

Competing interests

The authors declare no competing interests.

Ethical approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fan, L., Wang, B. Underwater image enhancement using scale-patch synergy transformer. SIViP 18, 3411–3420 (2024). https://doi.org/10.1007/s11760-024-03004-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-024-03004-8

Keywords

Search

Navigation