Elsevier

Neurocomputing

Volume 423, 29 January 2021, Pages 620-638
Neurocomputing

Prior guided conditional generative adversarial network for single image dehazing

https://doi.org/10.1016/j.neucom.2020.10.061Get rights and content

Abstract

Single image dehazing is an important problem as the existence of haze degrades the quality of the image and hinders most high-level computer vision tasks. Previous methods solve this problem using various low-level statistics priors or learning on synthetic data sets with CNN. In practice, the low-level priors are not always held in various scenes. And many CNN based methods directly estimate the transmission maps and atmospheric lights from huge synthetic data. However, without the guidance or constraints of priors may lead to over-dehazed or under-dehazed results. To address these issues, we propose a prior guided conditional generative adversarial network, an end-to-end model that generates realistic clean images using hazy image input and dehazed image based on the traditional prior-based method. The proposed generator extracted the feature with a parameters-shared encoder, and the clear image is recovered by decoding multi-scale features, which are fused and enhanced by the proposed attention-based feature aggregation block. And two-scale discriminators are adopted to supervise the generator to recover more image details with a combination of perceptual loss and adversarial loss. Our algorithm can efficiently combine the prior-based and CNN based image dehazing method and remove the weakness of each other. Experimental results on synthetic datasets and real-world images demonstrate our model can generate more perceptually appealing dehazing results, and provide superior performance compared with the state-of-the-art methods.

Introduction

Images are often degraded under bad weather condition especially in haze or fog environments. In the hazy condition, the reflected lights of the object are absorbed or scattered by the particles in the atmosphere, which cause the captured images to have low contrast and blur appearance [1]. Image dehazing aims to recovery the visual quality of the hazy images or improve the performance of high-level vision tasks such as classification, object detection, semantic segmentation and so on [2]. Hence, haze removal is essential for many vision applications. However, due to the complexity of the scene and environment, image dehazing faces great challenges, especially for the single image dehazing. It is an ill-posed inverse problem that requires to recover the unknown haze-free image, atmospheric light, and medium transmission from a single hazy image [3].

To resolve this problem, many algorithms were proposed by the computer vision community in recent years, which can be roughly divided into two categories: model-based methods and learning-based methods. Traditional model-based methods mainly relied on the low-level prior information, such as dark channel priors [4], color attenuation prior [5], haze-line prior [6]. These model-based methods are effective in single image dehazing due to the investigations of prior knowledge and understandings of the physical mechanism for hazes. However, these priors are mainly based on human observations and would not always hold for diverse real hazy scenes. Learning-based methods estimation the parameters from the training data. With the great success of deep learning technology in the field of computer vision, many deep learning-based single image dehazing methods were proposed to estimate the transmission map [7], [8], the atmosphere or both [9]. These methods have shown promising results for single image dehazing. However, the results of these methods are deteriorated by the inaccuracy of estimation images. Another kind of deep learning-based methods formulate the dehazing problem as image translation, and can directly recover the haze-free image without explicitly estimation transmission or atmosphere [10], [11]. However, these methods usually take CNNs to learn a mapping function from input hazy images to the hazy-free images, without considering haze-related priors to constrain the mapping space compared with the traditional methods. Furthermore, to learn the mapping function, many hazy and hazy-free images needed to synthesis according to the atmospheric scattering model [12]. Without any prior information on haze, the model may overfit to the training data and degrade the performance of dehazing, resulting in artifacts, color distortion, and insufficient haze removal in real hazy scenes. To address these problems, some methods introduce additional information to refine the estimation of atmosphere light, transmission maps or haze-free image, such as layer separation based multi-physical models [13], holistic edge [14], haze-relevant image priors [15], [16], [17] and multi-derived images [18], [19]. The key of these methods is what kinds of information being used to guide the image dehazing and how to fuse the additional information which is not always correct.

In this paper, we propose a new paradigm for single image dehazing based on a prior information guided conditional generative adversarial network(cGAN). The network contains an encoder-decoder based generator to recover hazy-free image, and a multi-scale discriminator to determines the authenticity of the generated image. Although any kinds of image-based prior information can be used as guidance, such as prior-based dehazed image, edge, or semantic map of the hazy image, etc., this paper mainly focused on using the prior-based dehazed image (dehazed image based on [4]) as guidance information. Since the prior image may contain some artifacts, the positive information selection is of vital importance for the guidance-based image dehazing. To extract useful feature of the prior image in data-driven based approach, we adopt an attention-based feature enhancement module to fuse different scales feature from the hazy image and the prior image. And to further guided the haze-free image recovering, an adaptive group-norm is adopted in the decoding phase. Finally, under the supervision of multi-scale discriminator, the generator can recover more details in the haze-free image. With the additional prior information, the conditional generative adversarial network (cGAN) can learn effective mapping function which is well adapted to indoor and outdoor scenes. On the other hand, the artifacts of the traditional prior-based method can be removed by the CNN. We evaluate the proposed algorithm against the state-of-the-art methods on numerous datasets comprised of synthetic and real-world hazy images.

Our contributions are summarized as follows:

  • We propose a new paradigm for single image dehazing based on prior image guided cGAN. The positive information of the prior based dehazed image is extracted by an attention-based feature aggregation module. And an adaptive group-norm module is proposed for aiding the partially correct style transfer in the decoding phase. This structure can effectively improve the visual effect of the recovered image.

  • We developed a two-scale discriminator to supervise the generator to recover high-resolution image with more details. A random crop patch from the recovered image is adopted to discriminate the realness, which can make the generator pay more attention to the local variations in the processes of learning.

  • We adopted a combined loss function that contains the structure similarity loss, pixel-wise loss, and adversarial loss. Using this objective function to optimize the network, the model can provide more perceptual dehazing images. Extensive experimental studies on different datasets including synthetic and real-world data have verified the effectiveness of the proposed method.

  • We analyze the difference of various prior guidance and fusing methods, and shown that our approach performs favorably against the state-of-the-arts both in indoor and outdoor scenes even trained with only indoor samples.

In this paper, we extend our preliminary work [15] in three aspects. First, we redesign the haze-free image generator by proposing the attention-based feature aggregation and adaptive group-norm module (Section 3.2) while the performance is obviously improved. Second, we develop an enhanced multi-scale discriminator (Section 3.3) and adopt a combined loss function for image dehazing refinement (Section 3.4). Third, we present more technical details, performance evaluation and quantitative analysis of the proposed algorithm.

The remainder of this paper is organized as follows. In Section 2, we briefly review generative adversarial networks and existing dehazing methods, such as model-based and deep learning-based methods. In Section 3, the details of the proposed prior guided cGAN and the design of our dehazing network are illustrated. The proposed method is analyzed and evaluated comprehensively, including the experimental setup, implemented details, ablation studies, subjective and objective evaluation results are presented in Section 4. Finally, the discussion and conclusion are presented in 5 Analysis and discussions, 6 Conclusion.

Section snippets

Related work

According to the generation process of the image, Narasimhan et.al [20] proposed the widely used atmosphere scattering model to describe the hazy, which can be formulated as:I(x)=J(x)·t(x)+(1-t(x))Awhere the I(x) is the captured hazy image, J(x) is the corresponding clean image, Ais the global atmosphere light, t(x) is the medium transmission map indicating the portion of the light which is not scattered and reaches the camera. When the image is homogeneous, the transmission map t(x) can be

Proposed prior guided dehazing methods

In this section, we analyze the character of prior based dehazing methods from the visual effects and feature response (Section 3.1). Based on the observation, we modify the conditional GAN framework by utilizing the guidance of prior-based dehazed results to generate clear images from hazy inputs. For simplicity and efficiency, we use the dark channel prior based dehazed image as the additional guidance. Our network can generate satisfactory hazy-free images even with the partially correct

Experiments

In Section 4.1, we first describe the datasets and the implementation details of the experiments. And in Section 4.2, we adopt the synthetic and real-world datasets to evaluate the performance of the proposed algorithm qualitatively and quantitatively by comparison with eleven state-of-the-art dehazing methods respectively. For the fairness of comparison, we adopt the source codes of the compared methods which are presented publicly by the authors. And we do not finetune or retrain except on

Effects of different prior information

As shown in 3 Proposed prior guided dehazing methods, 4 Experiments, our proposed paradigm for single image dehazing based on a prior information guided conditional generative adversarial network(cGAN) is generic to many kinds of prior image and performs favorably against the state-of-the-art image dehazing methods. In the following, we analyze the effects of different image-based prior information.

We adopt three kinds of prior image as guidance in our network: (1) White balanced image (WB)

Conclusion

In this paper, we propose prior guided cGAN for single image dehazing which does not rely on the estimations of the transmission map and atmospheric light. And we transform the problem of image dehazing to the problem of conditional image generation with the guidance of prior-based dehazed image. Our generator adopts an encoder-decoder architecture to directly generate haze-free images. Using the proposed attention-based feature aggregation module, the encoder can efficiently extract and fuse

CRediT authorship contribution statement

Yan Zhao Su: Conceptualization, Methodology, Writing - original draft, Software, Validation. Zhi Gao Cui: Visualization, Formal analysis, Writing - review & editing. Chuan He: Investigation, Funding acquisition. Ai Hua Li: Conceptualization, Supervision. Tao Wang: Project administration, Supervision. Kun Cheng: Project administration, Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grants 61773389, the Postdoctoral Science Foundation of China under Grants 2019 M663635, and the Natural Science Foundation of Shaanxi Province under Grant 2020JQ-298.

Yanzhao Su received the B.S., M.S., and Ph.D. degrees from the High-tech Institute of Xi'an, China, in 2008, 2010, and 2015, respectively. Currently he is a lecture with High-tech Institute of Xi'an. His research interests include image processing, computer vision, pattern recognition, and machine learning.

References (53)

  • W.C. Wang et al.

    Dehazing for images with large sky region

    Neurocomputing

    (2017)
  • C.O. Ancuti et al.

    Single image dehazing by multi-scale fusion

    IEEE Trans. Image Process.

    (2013)
  • D. Guo et al.

    Degraded image semantic segmentation with dense-gram networks

    IEEE Trans. Image Process.

    (2019)
  • R.T. Tan

    Visibility in bad weather from a single image

  • K. He et al.

    Single image haze removal using dark channel prior

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2011)
  • Q. Zhu et al.

    A fast single image haze removal algorithm using color attenuation prior

    IEEE Trans. Image Process.

    (2015)
  • R. Fattal Dehazing using color-lines ACM Trans. Graph. TOG, 34(1) (2014)...
  • W. Ren et al.

    Single image dehazing via multi-scale convolutional neural networks

  • B. Cai et al.

    Dehazenet: An end-to-end system for single image haze removal

    IEEE Trans. Image Process.

    (2016)
  • H. Zhang et al.

    Densely connected pyramid dehazing network

  • X. Liu et al.

    GridDehazeNet: attention-based multi-scale network for image dehazing

  • Xu Qin,Zhilin Wang ,Yuanchao Bai,Xiaodong Xie,Huizhu Jia. FFA-Net: Feature Fusion Attention Network for Single Image...
  • B. Li et al.

    Benchmarking single-image dehazing and beyond

    IEEE Trans. Image Process.

    (2019)
  • Z. Deng, L. Zhu, X. Hu, and C. Fu. Deep multi-model fusion for single-image dehazing, in: Proceeding of IEEE...
  • W. Ren et al.

    Single image dehazing via multi-scale convolutional neural networks with holistic edges

    Int. J. Comput. Vision

    (2020)
  • Y.Z. Su et al.

    Dark channel prior guided conditional generative adversarial network for single image dehazing

  • D. Yang, J. Sun. Proximal Dehaze-Net: a prior learning-based deep network for single image dehazing, in: Proceedings of...
  • Y. Liu. J. Pan, J. Ren, and Z. Su. Learning deep priors for image dehazing, in: Proceeding of IEEE International...
  • W. Ren et al.

    Gated fusion network for single image dehazing

  • F. Guo, X. Zhao, J. Tang, H. Peng, L.J. Liu, B.J. Zou, Single image dehazing based on fusion strategy, Neurocomputing,...
  • S.G. Narasimhan et al.

    Contrast restoration of weather degraded images

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2003)
  • R. Fattal

    Single image dehazing

    ACM Trans. Graphics (TOG)

    (2008)
  • J.B. Wang, N. He, L.L. Zhang, K. Lu. Single image dehazing with a physical model and dark channel prior,...
  • D. Berman et al.

    Non-local image dehazing

  • T.M. Bui et al.

    Single Image Dehazing using color ellipsoid prior

    IEEE Trans. Image Process.

    (2018)
  • Y. Zhang et al.

    Single image numerical iterative dehazing method based on local physical features[J]

    IEEE Trans. Circuits Syst. Video Technol.

    (2019)
  • Cited by (18)

    • Image dehazing via enhancement, restoration, and fusion: A survey

      2022, Information Fusion
      Citation Excerpt :

      Qin et al. [107] proposed a feature fusion attention module, where features from different stages are fused then reweighted both channel-wisely and spatial-wisely. Similar design is also adopted in [117] and [163]. The image-level fusion and feature-level fusion have a difference in application scenarios.

    • CHC-UNet: Cascaded Heterogeneous Convolution UNet Framework for Adaptive Image Dehazing

      2023, International Journal of Electrical and Electronic Engineering and Telecommunications
    View all citing articles on Scopus

    Yanzhao Su received the B.S., M.S., and Ph.D. degrees from the High-tech Institute of Xi'an, China, in 2008, 2010, and 2015, respectively. Currently he is a lecture with High-tech Institute of Xi'an. His research interests include image processing, computer vision, pattern recognition, and machine learning.

    Zhigao Cui received B.S. degree and M.S. degree both from the Xi’an research institute of High-Tech, Xi’an, China, in 2007 and 2010, respectively. And he received the Ph.D. degree in Tsinghua University, Beijing, China, in 2014. Now he is an Associate Professor in the Xi’an research institute of High-Tech, Xi’an, China. His research interests are computer vision, pattern recognition and video surveillance. By now, He has published more than 10 papers in journals which can be indexed by SCI and EI.

    Chuan He received the B.S., M.S., and Ph.D. degrees from the High-tech Institute of Xi'an, China, in 2008, 2010, and 2015, respectively. He is currently an Associate Professor with High-tech Institute of Xi'an. He has authored or coauthored two books and more than ten articles. His research interests include image processing, video metrics, machine learning, synthetic aperture radar image interpretation, pattern recognition, and deep learning.

    Aihua Li received the Ph.D. degrees from the Xi'an Jiao tong University, Shan Xi, China, in 1998. He is currently a Professor with High-tech Institute of Xi'an. His main research interests include machine vision, pattern recognition, mechanical fault diagnosis, and artificial intelligence.

    Tao Wang received the B.S., M.S., and Ph.D. degrees from the High-tech Institute of Xi'an, China, in 1996, 2003, and 2012. He is currently a Professor with High-tech Institute of Xi'an. His main research interests include computer vision, pattern recognition, mechanical fault diagnosis, and Hyperspectral image processing.

    Kun Cheng received the B.S. and M.S. degrees from the High-tech Institute of Xi'an, China, in 2007, and 2009. He is currently a researcher with High-tech Institute of Xi'an. His main research interests include computer vision, pattern recognition, mechanical fault diagnosis and big data analysis.

    View full text