Tricolor Pre-equalization Deblurring for Underwater Image Enhancement

Abstract

To enhance the qualitative and quantitative performance of underwater images, this paper proposes a tricolor pre-equalization deblurring method. In the proposed methodology, the tricolor histogram equalization is firstly used to change the level of chroma, contrast and intensity of underwater images. Then, the tricolor background light can be robustly estimated by using the dark channel prior. Finally, the tricolor transmission map is estimated by optical properties of underwater imagery. As compared with other enhancement methods, the experimental results demonstrate that the proposed method can significantly improve visual quality of underwater imagery, and quantifiably enhance the objective quality of underwater images. Moreover, the proposed method obtains the moderate complexity performance.

1 Introduction

With the resource exploration inside seas, lakes and rivers, underwater imagery has become an important research field. But, the problem with underwater scenario is the loss of colors and contrast in an image [1]. Underwater images often suffer from color distortion and low contrast because light is scattered and absorbed when traveling through water. An underwater image can be expressed as a linear superposition of a forward scattering component and a back scattering component. Such a forward scattering results in the blurring of image features whereas the back scattering obscures the details of the scene. Because each color differ in wavelength and energy level, every color absorbs at a different rate. The reason of most underwater images show green and/or blue in color is that the orange or red lights which has longer wavelengths are absorbed more quickly. Thus, underwater images usually perform predominantly in blue-green hue. As shown in Fig. 1, the forward scattering causes the blur degradation, and the back scattering causes the contrast degradation of underwater optical imaging. The underwater image is blurred from the actual characteristics, mainly caused by forward scattering of the light. While the back scattering actually tends to make the misty and foggy appearance of the distant object in the image and the scene has poor contrast. The capability to fully extract valuable information from underwater images for further processing such as aquatic robot inspection and marine mine detection is deteriorated by the overall poor visibility. So, enhancing such underwater images is a valuable work. The recent reviews of underwater image enhancement can be found in [2, 3].

Fig. 1.

The forward-scattering component and back-scattering component in underwater optical imaging (Color figure online).

1.1 Related Works

The enhancement of underwater image is known to be an ill-posed problem. Some underwater image enhancement methods have been proposed mainly by (a) dehazing the image, (b) compensating non-uniform illumination, or (c) increasing the image contrast and correcting the color shift. Fattal [4] proposed a single image dehazing (SID) method for estimating the optical transmission in hazy scenes. Ground on this estimation, the scattered light is eliminated to increase scene visibility and recover blur-free scene contrasts. The SID method exploits the fact that the surface shading and transmission functions are locally statistically uncorrelated. Bianco et al. [5] presented a simple yet effective prior that utilizes the strong difference in attenuation among the three color channels in the water to estimate the depths of the underwater scene, which used a graph-cut method to refine the depth map of dark channel prior for obtaining the clear image. Chiang and Chen [6] improved underwater images by combining a dehazing method with wavelength compensation. He et al. [7] enhanced a dark channel prior (DCP) to remove blurry or foggy effects from the spoilt images. According to the amount of attenuation of each wavelength, reverse compensation is conducted to reduce the distortion from color cast. The defect of dark channel prior is to decrease the contrast and darken the resulting image in some situations. Ancuti et al. [8] enhanced the visual quality of underwater images and videos by using fusion principles. In the fusion-based method, various types of weight maps give us the enhancement of images with higher quality, but the image fusion can’t be achieved simultaneously using this method. Galdran et al. [9] proposed a red channel method, where the lost contrast and color associated with short wavelength are recovered. The red-channel restoration method can be regarded as a simple extension of atmosphere dark channel prior, and the experiment results show that this method is good in the artificial lighting field, where the color correction and visibility can been improved.

Adaptive histogram equalization (AHE) is a typical technique which is used in image processing to enhance the contrast of images. AHE is different from ordinary histogram equalization. The adaptive method computes several histograms which respectively corresponds to a distinct section of an image, and utilizes them to redistribute the lightness values of the image. In this way, the local contrast can be improved. However, AHE has a tendency to overamplify the noise in relatively homogeneous regions of an image. A deviation of AHE called contrast limited adaptive histogram equalization (CLAHE) may avoid the tendency by limiting the amplification [10]. CLAHE is a generalization of adaptive histogram equalization where the contrast of an image are kept. The CLAHE model is originally developed for the enhancement of images with low contrast, and operates on the tiles of an image. Tiles are the small regions in the image which is divided according to a particular grid to exploit local spatial coherence in the scene. CLAHE enhances the contrast of each tile. To eliminate the induced artificial boundaries, the neighboring tiles are combined using bilinear interpolation. The contrast became limited to avoid amplifying any noise especially in homogeneous areas of an image. So, CLAHE limits the amplification by clipping the histogram at a user-defined value called clip limit. The probability-based (PB) method [11] is another image enhancement mechanism with simultaneous illumination and reflectance estimation, which is often used to enhance underwater images in related literatures.

Generally, the current deblurring methods unveil limited details and color of underwater images under several challenging scenes with the limited visible light, and they difficultly remove the effects of noise. For underwater image enhancement, the above methods possibly emphasize one aspect of either qualitative quality or quantitative quality, and ignore the comprehensive evaluation. In this work, the proposed method intends to address the above-mentioned problems.

1.2 Proposed Research

During underwater imagery, the scattering effect of light in the water causes the blur of the image. If the blurry image has a larger background area and low contrast, some deblurring methods possibly cause bad results as well as reduce the contrast of foreground. The existing methods have deserted the use of gamma correction and histogram stretching to reduce the noise problem which will be presented in the output image of the blur removal methods. In this work, we will propose a tricolor pre-equalization deblurring (TPD) method to remove foggy/hazy appearance in an underwater image. We firstly apply a histogram equalization technique with a color correction, and then optimize the deblurring mechanism by improving background light estimation and transmission map estimation, so as to obtain better visual results and to increase the objective quality and complexity performance.

1.3 Paper Organization

The rest of this paper is organized as follows. Section 2 introduces the proposed tricolor pre-equalization deblurring method. Section 3 evaluates and compares different enhancement methods’ experimental results. Finally, Sect. 4 concludes this paper and discusses future works.

2 Tricolor Pre-equalization Deblurring

For underwater image enhancement, this section will review and summarize the tricolor pre-equalization deblurring (TPD) method by using the principle of contrast limited adaptive histogram equalization and dark channel prior. Figure 2 shows the module diagram of the proposed TPD method. First of all, underwater images are pre-processed by tricolor histogram equalization, and then enhanced through tricolor dark channel prior mechanism.

Fig. 2.

Module diagram of the tricolor pre-equalization deblurring method.

Underwater lighting conditions are very complicated, and the color and contrast of underwater imagery undergoes a strong color-dependent attenuation. Following the previous research, a tricolor underwater imagery model can be represented as follows:

$$ \varvec{I}^{c} (x) = \varvec{J}^{c} (x) \varvec{t}^{c} (x) + \varvec{L}^{c} [1 - \varvec{t}^{c} (x)],c \in \{ r,g,b\} $$

(1)

where $x$ is a pixel for each color-component image; $\varvec{I}^{c} (x)$ is the blur-mixed intensity of an observed image; $\varvec{J}^{c} (x)$ is the recovered scene; $\varvec{L}^{c}$ is the background light that represents the contribution from the backscattering effect; $\varvec{t}^{c} (x) \in [0,1]$ is the transmission map and it is used to describe the portion of the light which does not reach and scatter the camera. Again, $1 - \varvec{t}^{c} (x)$ represents the thickness of blur. Thus, $\varvec{J}^{c} (x) \varvec{t}^{c} (x)$ and $\varvec{A}^{c} [1 - \varvec{t}^{c} (x)]$ denote the forward scattered component and backscattered component in underwater optical imaging. The purpose of deblurring is to recover $\varvec{J}^{c} (x)$, $\varvec{L}^{c}$, and $\varvec{t}^{c} (x)$ from $\varvec{I}^{c} (x)$. $\varvec{t}^{c} (x)$ represents the percentage of residual energy when the foreground irradiance passes through the medium. Since $\varvec{I}^{c} (x)$ is the intensity of the actual image mixed with background light, $\varvec{I}^{c} (x)$ is usually brighter than $\varvec{J}^{c} (x)$, with a low value namely as the transmission map $\varvec{t}^{c} (x)$. So, the dark channel of $\varvec{I}^{c} (x)$ has a high value as compare with $\varvec{J}^{c} (x)$ and that is the distinction which helps to remove blur. To solve this kind of ill-posed problem, the proposed TPD method includes the following main steps.

2.1 Tricolor Histogram Equalization

Firstly, we take the physical spectral characteristics-based color correction. In this work, we improve the CLAHE method [10], where two different priors (local contrast and color) are combined. To remove the limitation of dark channel prior, we add a histogram equalization process before tricolor dark channel prior. In this process, the resulting image is divided into each channel and enhanced by adaptive histogram equalization. Then, the result is processed through a color correction technique which is the refined image. For color correction, the mean value and the mean square error are computed in RGB channels of original image. Then, the maximum and minimum of each channel is calculated by

$$ \begin{array}{*{20}c} {\varvec{I}_{\hbox{max} }^{c} = \varvec{I}_{mean}^{c} + \mu \varvec{I}_{\text{var}}^{c} } \\ {\varvec{I}_{\hbox{min} }^{c} = \varvec{I}_{mean}^{c} - \mu \varvec{I}_{\text{var}}^{c} } \\ \end{array} $$

(2)

where $c \in \{ r,g,b\}$; $\varvec{I}_{mean}^{c}$ and $\varvec{I}_{\text{var}}^{c}$ are the mean value and the mean square error (MSE) in the RGB channel, respectively; $\mu$ is a parameter to control the image variation. Finally, the color-corrected image is obtained by

$$ \varvec{I}_{CR}^{c} = \frac{{\varvec{I}^{c} - \varvec{I}_{\hbox{min} }^{c} }}{{\varvec{I}_{\hbox{max} }^{c} - \varvec{I}_{\hbox{min} }^{c} }} \times 255 \times \alpha_{c} $$

(3)

where $\varvec{I}_{CR}^{c}$ is single-color enhanced image, and $\varvec{I}^{c}$ is single-color original image, and $\alpha_{c}$ is a weighting coefficient. The proposed TPD method is based on the statistics of histogram distribution of visually appealing natural-scene images. Intuitively, the histogram distributions of natural-scene images are wider and more consistent while the histogram distribution of each color channel of underwater image is shifted in a horizontal direction due to the effects of the absorption and scattering as well as the floating particles. The histogram of blue component concentrates on a brightest side, followed by the green component and then the red component. The histogram distributions of the contrast enhanced underwater image become wider and more consistent than those of the raw underwater image [12].

2.2 Calculating the Dark Channel

To robustly estimate the background light, we use a hierarchical searching technique, then remove the effects of suspended particles via the dark channel prior [7], and finally remove the disturbance of bright objects and determine the background light according to the properties of light travelling in the water. Dark channel prior is usually used to produce a natural blur-free image. However, we use this method to enhance underwater image. The presence of water particles and light scattering causes the blur in underwater images which can be removed by dark channel prior. The dark channel prior is used to remove blur from a single original image. It refers to the following observation on those images which don’t blur: in most of the non-lightsource region, the intensity value of at least one color channel shows very low at some pixels. Namely, the minimum intensity in such a patch has a very low value. The main aim of the blur removal method is the estimation of $\varvec{J}^{c} (x)$, $\varvec{t}^{c} (x)$, $\varvec{L}^{c}$. The dark channel prior shows that the performance of most of the local regions which present in the background of the image, is consistent with the blur-free images. $\varvec{J}_{dark} (x)$ represents the dark channel at $x$. Formally, for an image, its underwater dark channel prior can be defined as:

$$ \varvec{J}_{dark} (x) = \mathop {\hbox{min} }\limits_{{c \in \{ r,g,b\} }} \left( {{ \hbox{min} }_{{y \in\Omega (x)}} \varvec{J}^{c} (y)} \right) $$

(4)

In above equation, $\varvec{J}^{c} (y)$ is one of the RGB channels of an underwater image, and ${\varvec{\Omega}}(x)$ is a square region (local patch, 15 × 15 pixels) centered at $x$. If $x$ doesn’t belong to local regions, then $\varvec{J}_{dark} (x)$ is low and tends to be zero. Except for the lightsource patches, the intensity of $\varvec{J}_{dark} (x)$ is showing a low value and tending to zero when $\varvec{J}^{c} (y)$ is a blur-free image. And the above knowledge or statistical observation is called the dark channel prior.

2.3 Estimating the Background Light

In most of the previous methods, the background light $\varvec{L}^{c}$ is estimated from the most blur-opaque pixel. For example, the pixel with highest intensity is used as the background light [4]. But in real images, the brightest pixel could on a white region. The dark channel of a blurry image approximates the blur denseness well. We can use the dark channel to improve the background light estimation. We first pick the top 0.1% brightest pixels in the dark channel. These pixels are most blur-opaque. Among these pixels, we select some pixels with highest intensity in the original image as the background light. Note that these pixels may not be brightest in the whole image. This method based on the dark channel prior is more robust and simple than the “brightest pixel” method. It is used to automatically evaluate the background lights for each image shown in this work. Based on the background light $\varvec{L}^{c}$, the transmission map is calculated by dividing the Eq. (1) by $\varvec{L}^{c}$. According to the dark channel prior, the dark channel of image without blur tends to zero [7], and He et al. provided the same transmission map for each color component:

$$ t(x) = 1 - \omega \mathop {\hbox{min} }\limits_{{c \in \{ r, g, b\} }} \left( {{ \hbox{min} }_{{y \in\Omega (x)}} \frac{{\varvec{I}^{c} (y)}}{{\varvec{L}^{c} }}} \right) $$

(5)

where the parameter $\omega (\omega = 0.9)$ keeps a small amount of blur in the image to perceive the depth of image. The above method didn’t consider the difference of three color component. In our method, each color component has its transmission map $t^{c} (x)$ as follows:

$$ \varvec{t}^{c} (x) = 1 - \frac{{\beta_{c} \mathop {\hbox{min} }\limits_{{c \in \{ r, g, b\} }} \left( {\min_{{y \in\Omega (x)}} \varvec{I}^{c} (y)} \right)}}{{\varvec{L}^{c} }} $$

(6)

where $\beta_{c} (\beta_{r} = 1.0,\beta_{g} = 0.9,\beta_{b} = 0.95)$ is color-aware parameter. We adopt a guided image filtering method to refine the transmission map.

2.4 Refining the Transmission Map

Based on the observation that the dark and bright regions of underwater images become too dark or too bright after being enhanced by the proposed TPD method, a filtered transmission map is employed to adjust the results for better visual quality. After obtaining the transmission map block by block, we incorporate the guided filter to refine the transmission map, because the block-based transmission map usually yields blocking artifacts. By replacing soft matting [13], a guided filter [14] is applied to refine the transmission map, and to find the accurate transmission map.

2.5 Recovering the Scene Radiance

With the transmission map, the scene radiance can be recovered by the proposed TPD method according to Eq. (1). However, if the transmission $\varvec{t}^{c} (x)$ is close to zero, the direct attenuation term $\varvec{J}^{c} (x)\varvec{t}^{c} (x)$ will approximate to zero [15]. Noise will easily appear in the directly recovered scene $\varvec{J}^{c} (x)$. Based on this, we place restrictions on the transmission $\varvec{t}^{c} (x)$ to a lower bound $t_{0}$, therefore, a small certain amount of blur are preserved in very dense blurry regions. The goal of blur removal is to recover $\varvec{J}^{c} (x)$, $\varvec{L}^{c}$, and $\varvec{t}^{c} (x)$ from $\varvec{I}^{c} (x)$. Using the blur imaging equation and the dark channel prior together, the recovered scene can be represented by:

$$ \varvec{J}^{c} (x) = \frac{{\varvec{I}^{c} (x) - \varvec{L}^{c} }}{{\hbox{max} (\varvec{t}^{c} (x),t_{0} )}} + \varvec{L}^{c} ,\;c \in \{ r,g,b\} $$

(7)

where $t_{0}$ is a threshold value to avoid a low value of denominator, and $t_{0}$ is usually set to 0.1. Because the brightness of the scene is usually less bright than the background light, the image looks dim after blur removal. So, we increase the exposure of $\varvec{J}^{c} (x)$ for display. The dark channel prior is effective for a variety of hazy images, however, it may be invalid if the scene objects are inherently similar to the background light and no shadow is cast on them. The underwater images are similar with the blurry images as they are all degraded through the medium. Besides, they doesn’t conform the failure condition [16]. Therefore, dark channel prior can be used to remove the blur in underwater images.

3 Experimental Results

To evaluate the proposed tricolor pre-equalization deblurring (TPD) method, extensive experiments are carried out. Without loss of generality, the underwater image enhancement methods include histogram equalization (HistEqu), single image dehazing (SID) [4], dark channel prior (DCP) [7], contrast limited adaptive histogram equalization (CLAHE) [10], Probability-based (PB) [11]. We used their Matlab codes to obtain different experimental results. To robustly compare the performance of different methods, we extracted the typical scenes which are also used by previous literatures, as shown in Fig. 3. For different underwater image enhancement methods, we present the comprehensive evaluation of qualitative quality, quantitative quality and runtime complexity.

Fig. 3.

Typical underwater images: (a) reef1, 500 × 375; (b) reef2, 750 × 1000; (c) reef3, 1000 × 707; (d) ocean2, 550 × 412; (e) Galdran_Im1, 473 × 353; (f) fish, 512 × 384; (g) Eustice4, 690 × 560; (h) Ancuti1, 404 × 303, (i) Ancuti2, 1037 × 778; (j) Ancuti3, 512 × 384.

3.1 Qualitative Comparison

3.1.1 Typical Scenes

For different methods, Fig. 4 gives an example of qualitative quality comparisons. As can been observed, the appearance of some methods is either over-enhanced or under-enhanced. The HistEqu method introduces artifacts due to ignoring the spatially varying distance dependencies [17]. Although the contrast and details are increased by DCP and PB methods, the colors and visibility are poor because the attenuated energy is not compensated individually based on different wavelengths [18]. Our TPD method has successfully enhanced the contrast, relatively genuine color, and visibility of the original underwater images.

Fig. 4.

Qualitative quality comparisons for the image Galdran_Im1: (a) Original underwater image; (b) HistEqu; (c) CLAHE; (d) SID; (e) DCP; (f) PB; (g) our TPD method.

3.1.2 Color Accuracy

Figure 5 shows the color-card example. Figure 5(a) shows the cards in bright sunlight with the spectral colors, and Fig. 5(b) shows the same cards at 60 ft in the Gulf of Mexico, and this photo is straight out of the camera with no filters or adjustments for white-balance, color, etc. As you can see, the spectral red is completely gone and difficult to distinguish from black. The orange now looks drab and almost an olive-green, and yellow holds fairly true, but green is now looking closer to yellow. Blue and indigo are OK, but contrasting with black, violet is similar to red. After enhancing the original photo by different methods, the result of our TPD method is more visible and has fewer color loss than the results of the compared methods.

Fig. 5.

Color accuracy test. (a) The standard ColorChecker cards image; (b) The underwater ColorChecker cards image; (c) HistEqu; (d) CLAHE; (e) SID; (f) DCP; (g) PB; (h) our TPD method (Color figure online).

3.2 Quantitative Comparison

Following previous literatures, information Entropy and patch-based contrast quality index (PCQI) [19] are employed to evaluate the no-reference image quality of the proposed TPD method. The higher Entropy values indicate that the enhancement method can sufficiently reduce information loss of restoring the underwater images and increase the valuable information. The higher PCQI values indicate the enhanced results can well balance the chroma, saturation, and contrast of the enhanced underwater images [20]. Figure 6 gives the average values of the Entropy and PCQI for all test images. Our TPD method outperforms other methods in terms of the Entropy values. Our TPD method ranks first in terms of PCQI evaluation. Since all enhancement methods use similar basis instructions, their MATLAB implementations can provide a certain reference for the complexity evaluation. As can been seen, the proposed TPD method obtains the moderate complexity performance.

Fig. 6.

Quantitative performance comparisons of different methods. (a) Entropy quality; (b) PCQI quality; (c) Runtime complexity.

4 Conclusion

In this work, we develop a physics-based image enhancement method for recovering visibility and colors of the degraded underwater images, where the tricolor histogram equalization followed by dark channel prior has been used to enhance underwater images. This proposed method is analyzed and compared with different methods. Experimental results show that the proposed method can better enhance underwater images, even for the images captured in the challenging underwater scenes. As a future work, we think that deep-learning approaches can be very valuable since it is difficult to empirically design such many priors and parameters for underwater image restoration.