research-article

Leveraging Deep Statistics for Underwater Image Enhancement

Authors:

Zheng-Jun ZhaAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 17, Issue 3s

Article No.: 116, Pages 1 - 20

https://doi.org/10.1145/3489520

Published: 26 October 2021 Publication History

Abstract

Underwater imaging often suffers from color cast and contrast degradation due to range-dependent medium absorption and light scattering. Introducing image statistics as prior has been proved to be an effective solution for underwater image enhancement. However, relative to the modal divergence of light propagation and underwater scenery, the existing methods are limited in representing the inherent statistics of underwater images resulting in color artifacts and haze residuals. To address this problem, this article proposes a convolutional neural network (CNN)-based framework to learn hierarchical statistical features related to color cast and contrast degradation and to leverage them for underwater image enhancement. Specifically, a pixel disruption strategy is first proposed to suppress intrinsic colors’ influence and facilitate modeling a unified statistical representation of underwater image. Then, considering the local variation of depth of field, two parallel sub-networks: Color Correction Network (CC-Net) and Contrast Enhancement Network (CE-Net) are presented. The CC-Net and CE-Net can generate pixel-wise color cast and transmission map and achieve spatial-varied color correction and contrast enhancement. Moreover, to address the issue of insufficient training data, an imaging model-based synthesis method that incorporates pixel disruption strategy is presented to generate underwater patches with global degradation consistency. Quantitative and subjective evaluations demonstrate that our proposed method achieves state-of-the-art performance.

References

[1]

Junku Yuh and Michael West. 2001. Underwater robotics. Adv. Robot. 15, 5 (2001), 609–639.

[2]

Katia Lebart, Chris Smith, Emanuele Trucco, and David M. Lane. 2003. Automatic indexing of underwater survey video: Algorithm and benchmarking method. IEEE J. Ocean. Eng. 28, 4 (2003), 673–686.

[3]

John Y. Chiang and Ying-Ching Chen. 2012. Underwater image enhancement by wavelength compensation and dehazing. IEEE Trans. Image Process. 21, 4 (2012), 1756–1769.

Digital Library

[4]

Matthew Johnson-Roberson, Mitch Bryson, Ariell Friedman, Oscar Pizarro, Giancarlo Troni, Paul Ozog, and Jon C. Henderson. 2017. High-resolution underwater robotic vision-based mapping and three-dimensional reconstruction for archaeology. J. Field Robot. 34, 4 (2017), 625–643.

[5]

Adriana Olmos and Emanuele Trucco. 2002. Detecting man-made objects in unconstrained subsea videos. In Proceedings of the 13th British Machine Vision Conference (BMVC’02). 1–10.

[6]

Gershon Buchsbaum. 1980. A spatial processor model for object colour perception. J. Franklin Inst. 310, 1 (1980), 1–26.

[7]

Kaiming He, Jian Sun, and Xiaoou Tang. 2010. Single image haze removal using dark channel prior. IEEE Trans. Pattern Anal. Mach. Intell. 33, 12 (2010), 2341–2353.

Digital Library

[8]

Haocheng Wen, Yonghong Tian, Tiejun Huang, and Wen Gao. 2013. Single underwater image enhancement with a new optical model. In Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS’13). IEEE, 753–756.

[9]

Paul Drews, Erickson Nascimento, Filipe Moraes, Silvia Botelho, and Mario Campos. 2013. Transmission estimation in underwater single images. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 825–830.

Digital Library

[10]

Pritish M. Uplavikar, Zhenyu Wu, and Zhangyang Wang. 2019. All-in-One underwater image enhancement using domain-adversarial learning. In Proceedings of the Computer Vision and Pattern Recognition Workshops (CVPR’19). 1–8.

[11]

Hanyu Li, Jingjing Li, and Wei Wang. 2019. A fusion adversarial underwater image enhancement network with a public test dataset. Retrieved from https://arXiv:1906.06819.

[12]

Jie Li, Katherine A Skinner, Ryan M. Eustice, and Matthew Johnson-Roberson. 2018. WaterGAN: Unsupervised generative network to enable real-time color correction of monocular underwater images. IEEE Robot. Autom. Lett. 3, 1 (2018), 387–394.

[13]

Young-Sik Shin, Younggun Cho, Gaurav Pandey, and Ayoung Kim. 2016. Estimation of ambient light and transmission map with common convolutional architecture. In Proceedings of the MTS/IEEE Oceans Conference (OCEANS’16). IEEE, 1–7.

[14]

Yang Wang, Jing Zhang, Yang Cao, and Zengfu Wang. 2017. A deep CNN method for underwater image enhancement. In Proceedings of the IEEE International Conference on Image Processing (ICIP’17). IEEE, 1382–1386.

[15]

Kaiming He, Jian Sun, and Xiaoou Tang. 2011. Single image haze removal using dark channel prior. IEEE Trans. Pattern Anal. Mach. Intell. 33, 12 (2011), 2341–2353.

Digital Library

[16]

L. Chao and M. Wang. 2010. Removal of water scattering. In Proceedings of the IEEE International Conference on Computer Engineering Technology (ICCET’10). 35–39.

[17]

Hung-Yu Yang, Pei-Yin Chen, Chien-Chuan Huang, Ya-Zhu Zhuang, and Yeu-Horng Shiau. 2011. Low complexity underwater image enhancement based on dark channel prior. In Proceedings of the 2nd International Conference on Innovations in Bio-inspired Computing and Applications (IBICA’11). IEEE, 17–20.

Digital Library

[18]

Xinwei Zhao, Tao Jin, and Song Qu. 2015. Deriving inherent optical properties from background color and underwater image enhancement. Ocean Eng. 94 (2015), 163–172.

[19]

Nicholas Carlevaris-Bianco, Anush Mohan, and Ryan M. Eustice. 2010. Initial results in underwater single image dehazing. In Proceedings of the MTS/IEEE Oceans Conference (OCEANS’10). IEEE, 1–8.

[20]

Codruta O. Ancuti, Cosmin Ancuti, Christophe De Vleeschouwer, and Rafael Garcia. 2017. Locally adaptive color correction for underwater image dehazing and matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 1–9.

[21]

Yang Wang, Yang Cao, Zheng-Jun Zha, Jing Zhang, Zhiwei Xiong, Wei Zhang, and Feng Wu. 2019. Progressive retinex: Mutually reinforced illumination-noise perception network for low-light image enhancement. In Proceedings of the 27th ACM International Conference on Multimedia. 2015–2023.

Digital Library

[22]

Bolun Cai, Xiangmin Xu, Kui Jia, Chunmei Qing, and Dacheng Tao. 2016. Dehazenet: An end-to-end system for single image haze removal. IEEE Trans. Image Process. 25, 11 (2016), 5187–5198.

Digital Library

[23]

Boyi Li, Xiulian Peng, Zhangyang Wang, Jizheng Xu, and Dan Feng. 2017. An all-in-one network for dehazing and beyond. Retrieved from https://arXiv:1707.06543.

[24]

Zhe Chen, Wanli Ouyang, Tongliang Liu, and Dacheng Tao. 2021. A shape transformation-based dataset augmentation framework for pedestrian detection. Int. J. Comput. Vision 129, 4 (2021), 1121–1138.

[25]

Yubin Deng, Chen Change Loy, and Xiaoou Tang. 2018. Aesthetic-driven image enhancement by adversarial learning. In Proceedings of the 26th ACM International Conference on Multimedia. 870–878.

Digital Library

[26]

Jing Zhang, Yang Cao, Zheng-Jun Zha, and Dacheng Tao. 2020. Nighttime dehazing with a synthetic benchmark. In Proceedings of the 28th ACM International Conference on Multimedia. 2355–2363.

Digital Library

[27]

Jing Zhang, Yang Cao, Shuai Fang, Yu Kang, and Chang Wen Chen. 2017. Fast haze removal for nighttime image using maximum reflectance prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7418–7426.

[28]

Jiawei Liu, Zheng-Jun Zha, Xuejin Chen, Zilei Wang, and Yongdong Zhang. 2019. Dense 3D-convolutional neural network for person re-identification in videos. ACM Trans. Multimedia Comput. Commun. Appl. 15, 1s (2019), 1–19.

Digital Library

[29]

Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, and Zheng-Jun Zha. 2020. Object relational graph with teacher-recommended learning for video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13278–13288.

[30]

Hongtao Xie, Shancheng Fang, Zheng-Jun Zha, Yating Yang, Yan Li, and Yongdong Zhang. 2019. Convolutional attention networks for scene text recognition. ACM Trans. Multimedia Comput. Commun. Appl. 15, 1s (2019), 1–17.

Digital Library

[31]

Yang Wang, Yang Cao, Zheng-Jun Zha, Jing Zhang, and Zhiwei Xiong. 2020. Deep degradation prior for low-quality image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11049–11058.

[32]

Chongyi Li, Jichang Guo, and Chunle Guo. 2018. Emerging from water: Underwater image color correction based on weakly supervised color transfer. IEEE Signal Process. Lett. 25, 3 (2018), 323–327.

[33]

Jing Zhang, Yang Cao, Yang Wang, Chenglin Wen, and Chang Wen Chen. 2018. Fully point-wise convolutional neural network for modeling statistical regularities in natural images. In Proceedings of the 26th ACM International Conference on Multimedia. 984–992.

Digital Library

[34]

Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. 2013. Maxout networks. Retrieved from https://arXiv:1302.4389.

Digital Library

[35]

Chongyi Li, Chunle Guo, Wenqi Ren, Runmin Cong, Junhui Hou, Sam Kwong, and Dacheng Tao. 2019. An underwater image enhancement benchmark dataset and beyond. IEEE Trans. Image Process. 29 (2019), 4376–4389.

[36]

Shijie Zhang, Jing Zhang, Shuai Fang, and Yang Cao. 2014. Underwater stereo image enhancement using a new physical model. In Proceedings of the IEEE International Conference on Image Processing (ICIP’14). IEEE, 5422–5426.

[37]

Dongliang Cheng, Dilip K. Prasad, and Michael S. Brown. 2014. Illuminant estimation for color constancy: Why spatial-domain methods work and the role of the color distribution. J. Optic. Soc. Amer. A 31, 5 (2014), 1049–1058.

[38]

Kaiming He, Jian Sun, and Xiaoou Tang. 2010. Guided image filtering. In Proceedings of the European Conference on Computer Vision. Springer, 1–14.

Digital Library

[39]

Edwin H. Land. 1977. The retinex theory of color vision. Sci. Amer. 237, 6 (1977), 108–129.

[40]

Joost Van De Weijer, Theo Gevers, and Arjan Gijsenij. 2007. Edge-based color constancy. IEEE Trans. Image Process. 16, 9 (2007), 2207–2214.

Digital Library

[41]

Graham D. Finlayson and Elisabetta Trezzi. 2004. Shades of gray and colour constancy. In Proceedings of the Color and Imaging Conference. Society for Imaging Science and Technology, 37–41.

[42]

Simone Bianco, Claudio Cusano, and Raimondo Schettini. 2015. Color constancy using CNNs. Retrieved from https://arXiv:1504.04548.

[43]

Chong-Yi Li, Ji-Chang Guo, Run-Min Cong, Yan-Wei Pang, and Bo Wang. 2016. Underwater image enhancement by dehazing with minimum information loss and histogram distribution prior. IEEE Trans. Image Process. 25, 12 (2016), 5664–5677.

Digital Library

[44]

Yan-Tsung Peng and Pamela C. Cosman. 2017. Underwater image restoration based on image blurriness and light absorption. IEEE Trans. Image Process. 26, 4 (2017), 1579–1594.

Digital Library

[45]

Codruta O. Ancuti, Cosmin Ancuti, Christophe De Vleeschouwer, and Philippe Bekaert. 2018. Color balance and fusion for underwater image enhancement. IEEE Trans. Image Process. 27, 1 (2018), 379–393.

[46]

Boyi Li, Xiulian Peng, Zhangyang Wang, Jizheng Xu, and Dan Feng. 2017. Aod-net: All-in-one dehazing network. In Proceedings of the IEEE International Conference on Computer Vision. 4770–4778.

[47]

J. Zhang and D. Tao. 2020. FAMED-Net: A fast and accurate multi-scale end-to-end dehazing network. IEEE Trans. Image Process.: Pub. IEEE Signal Process. Soc. 29 (2020), 72.

[48]

Daniel Scharstein and Richard Szeliski. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vision 47, 1–3 (2002), 7–42.

Digital Library

[49]

Daniel Scharstein and Richard Szeliski. 2003. High-accuracy stereo depth maps using structured light. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 1. IEEE, I–I.

Digital Library

[50]

Daniel Scharstein and Chris Pal. 2007. Learning conditional random fields for stereo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’07). IEEE, 1–8.

[51]

Yan-Tsung Peng and Pamela C. Cosman. 2017. Underwater image restoration based on image blurriness and light absorption. IEEE Trans. Image Process. 26, 4 (2017), 1579–1594.

Digital Library

[52]

Cosmin Ancuti, Codruta Orniana Ancuti, Tom Haber, and Philippe Bekaert. 2012. Enhancing underwater images and videos by fusion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). IEEE, 81–88.

Digital Library

[53]

Karen Panetta, Chen Gao, and Sos Agaian. 2015. Human-visual-system-inspired underwater image quality measures. IEEE J. Ocean. Eng. 41, 3 (2015), 541–551.

[54]

Derya Akkaynak and Tali Treibitz. 2019. Sea-thru: A method for removing water from underwater images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1682–1691.

[55]

Chongyi Li, Saeed Anwar, and Fatih Porikli. 2020. Underwater scene prior inspired deep underwater image and video enhancement. Pattern Recogn. 98 (2020), 107038.

Digital Library

[56]

Houwen Peng, Bing Li, Haibin Ling, Weiming Hu, Weihua Xiong, and Stephen J. Maybank. 2016. Salient object detection via structured matrix decomposition. IEEE Trans. Pattern Anal. Mach. Intell. 39, 4 (2016), 818–832.

Digital Library

[57]

Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. 2006. SURF: Speeded up robust features. In Proceedings of the European Conference on Computer Vision. Springer, 404–417.

Digital Library

[58]

Risheng Liu, Xin Fan, Ming Zhu, Minjun Hou, and Zhongxuan Luo. 2020. Real-world underwater enhancement: Challenges, benchmarks, and solutions under natural light. IEEE Trans. Circ. Syst. Video Technol. 30, 12 (2020), 4861–4875.

Digital Library

Cited By

Bi XWang PGuo WZha FSun L(2024)RGB/Event signal fusion framework for multi-degraded underwater image enhancementFrontiers in Marine Science10.3389/fmars.2024.136681511Online publication date: 30-May-2024
https://doi.org/10.3389/fmars.2024.1366815
Zhang YCai YYan DLin R(2024)Real-World Scene Image Enhancement with Contrastive Domain Adaptation LearningACM Transactions on Multimedia Computing, Communications, and Applications10.1145/369497320:12(1-23)Online publication date: 26-Nov-2024
https://dl.acm.org/doi/10.1145/3694973
Hsu WHsu Y(2024)Multi-Scale and Multi-Layer Lattice Transformer for Underwater Image EnhancementACM Transactions on Multimedia Computing, Communications, and Applications10.1145/368880220:11(1-24)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.1145/3688802
Show More Cited By

Index Terms

Leveraging Deep Statistics for Underwater Image Enhancement
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Reconstruction

Recommendations

Underwater image enhancement by color correction and color constancy via Retinex for detail preserving
Abstract
In underwater, light attenuation causes non-uniform illumination that degrades underwater image. To enhance the degraded image, we propose an underwater image enhancement method that includes color correction, color constancy, multi-...
Graphical abstract

Display Omitted
Highlights
- The formulation of color correction compensates the red and blue channels by masking.
Underwater image enhancement by combining color constancy and dehazing based on depth estimation
Highlights
- Proposed an underwater image enhancement method using color constancy and dehazing.
Abstract
The physical properties which are present in the underwater environment affects the images captured by the visual sensors. As a consequence of these properties, the captured image includes non-uniform illumination. This non-uniform ...
Underwater image enhancement method with light scattering characteristics
Highlights
- Underwater image is divided into five categories.
- Calculate the color loss rate ...
Abstract
Light is absorbed and scattered when propagating in water, which results in low quality and poor visibility of underwater optical images. Furthermore, the absorption of light by water causes color distortion, whereas the scattering of ...
Graphical abstract

Display Omitted

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 17, Issue 3s

October 2021

324 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3492435

Editor:
Alberto Del Bimbo
University of Firenze, Italy

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 October 2021

Accepted: 01 August 2021

Revised: 01 July 2021

Received: 01 December 2020

Published in TOMM Volume 17, Issue 3s

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

National Key R&D Program of China
National Natural Science Foundation of China (NSFC)
University Synergy Innovation Program of Anhui Province
Major Special Science and Technology Project of Anhui
key scientific technological innovation research project by Ministry of Education

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

72
Total Citations
View Citations
858
Total Downloads

Downloads (Last 12 months)96
Downloads (Last 6 weeks)9

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bi XWang PGuo WZha FSun L(2024)RGB/Event signal fusion framework for multi-degraded underwater image enhancementFrontiers in Marine Science10.3389/fmars.2024.136681511Online publication date: 30-May-2024
https://doi.org/10.3389/fmars.2024.1366815
Zhang YCai YYan DLin R(2024)Real-World Scene Image Enhancement with Contrastive Domain Adaptation LearningACM Transactions on Multimedia Computing, Communications, and Applications10.1145/369497320:12(1-23)Online publication date: 26-Nov-2024
https://dl.acm.org/doi/10.1145/3694973
Hsu WHsu Y(2024)Multi-Scale and Multi-Layer Lattice Transformer for Underwater Image EnhancementACM Transactions on Multimedia Computing, Communications, and Applications10.1145/368880220:11(1-24)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.1145/3688802
Liang XYang EDeng CYang Y(2024)CrossFormer: Cross-Modal Representation Learning via Heterogeneous Graph TransformerACM Transactions on Multimedia Computing, Communications, and Applications10.1145/368880120:12(1-21)Online publication date: 20-Sep-2024
https://dl.acm.org/doi/10.1145/3688801
Hsu CLi YTsai CWang JHsu C(2024)Federated Learning Using Multi-Modal Sensors with Heterogeneous Privacy Sensitivity LevelsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/368680120:11(1-27)Online publication date: 5-Aug-2024
https://dl.acm.org/doi/10.1145/3686801
Yang YQiu SZeng LPan Z(2024)Detail-preserving Joint Image UpsamplingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/366524620:8(1-23)Online publication date: 13-Jun-2024
https://dl.acm.org/doi/10.1145/3665246
Liu FYe MDu BCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Cloth-aware Augmentation for Cloth-generalized Person Re-identificationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680956(4053-4062)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680956
Syu JLin JSrivastava G(2024)Distributed Learning Mechanisms for Anomaly Detection in Privacy-Aware Energy Grid Management SystemsACM Transactions on Sensor Networks10.1145/3640341Online publication date: 17-Jan-2024
https://dl.acm.org/doi/10.1145/3640341
Xia SXing TWu CLiu GYang JLi K(2024)AQMon: A Fine-grained Air Quality Monitoring System Based on UAV Images for Smart CitiesACM Transactions on Sensor Networks10.1145/363876620:2(1-20)Online publication date: 19-Jan-2024
https://dl.acm.org/doi/10.1145/3638766
Yang SShen LShuai HFeng K(2024)CMAF: Cross-Modal Augmentation via Fusion for Underwater Acoustic Image RecognitionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363642720:5(1-25)Online publication date: 11-Jan-2024
https://dl.acm.org/doi/10.1145/3636427
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents