Skip to main content
SpringerLink
Log in

Single Image Dehazing via Multi-scale Convolutional Neural Networks with Holistic Edges

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Single image dehazing has been a challenging problem which aims to recover clear images from hazy ones. The performance of existing image dehazing methods is limited by hand-designed features and priors. In this paper, we propose a multi-scale deep neural network for single image dehazing by learning the mapping between hazy images and their transmission maps. The proposed algorithm consists of a coarse-scale net which predicts a holistic transmission map based on the entire image, and a fine-scale net which refines dehazed results locally. To train the multi-scale deep network, we synthesize a dataset comprised of hazy images and corresponding transmission maps based on the NYU Depth dataset. In addition, we propose a holistic edge guided network to refine edges of the estimated transmission map. Extensive experiments demonstrate that the proposed algorithm performs favorably against the state-of-the-art methods on both synthetic and real-world images in terms of quality and speed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

References

  • Ancuti, C. O., & Ancuti, C. (2013). Single image dehazing by multi-scale fusion. IEEE Transactions on Image Processing, 22(8), 3271–3282.

    Article  Google Scholar 

  • Ancuti, C. O., Ancuti, C., & Bekaert, P. (2010). Effective single image dehazing by fusion. In IEEE international conference on image processing, (pp. 3541–3544).

  • Berman, D., Treibitz, T., & Avidan, S. (2017). Air-light estimation using haze-lines. In IEEE international conference on computational photography, (pp. 1–9).

  • Berman, D., Treibitz, T., & Shai, A. (2016). Non-local image dehazing. In IEEE conference on computer vision and pattern recognition, (pp. 1674–1682).

  • Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32.

    Article  Google Scholar 

  • Cai, B., Xu, X., Jia, K., Qing, C., & Tao, D. (2016). Dehazenet: An end-to-end system for single image haze removal. IEEE Transactions on Image Processing, 25(11), 5187–5198.

    Article  MathSciNet  Google Scholar 

  • Cai, B., Xu, X., & Tao, D. (2016). Real-time video dehazing based on spatio-temporal mrf. In Pacific Rim conference on multimedia, (pp. 315–325).

  • Canny, J. (1986). A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8(6), 679–98.

    Article  Google Scholar 

  • Caraffa, L., & Tarel, J. P. (2012). Stereo reconstruction and contrast restoration in daytime fog. In Asian conference on computer vision, (pp. 13–25).

  • Caraffa, L., & Tarel, J. P. (2013). Markov random field model for single image defogging. In IEEE intelligent vehicles symposium, (pp. 994–999).

  • Dong, C., Loy, C. C., He, K., & Tang, X. (2014). Learning a deep convolutional network for image super-resolution. In European conference on computer vision, (pp. 184–199).

  • Eigen, D., Puhrsch, C., & Fergus, R. (2014). Depth map prediction from a single image using a multi-scale deep network. In Advances in neural information processing systems, (pp. 2366–2374).

  • Fattal, R. (2008). Single image dehazing. ACM Transactions on Graphics, 27(3), 72.

    Article  Google Scholar 

  • Fattal, R. (2014). Dehazing using color-lines. ACM Transactions on Graphics, 34(1), 13.

    Article  Google Scholar 

  • Geiger, A., Lenz, P., & Urtasun, R. (2012). Are we ready for autonomous driving? The kitti vision benchmark suite. In IEEE conference on computer vision and pattern recognition, (pp. 3354–3361).

  • Gibson, K. B., & Nguyen, T. Q. (2013). Fast single image fog removal using the adaptive wiener filter. In IEEE international conference on image processing, (pp. 714–718).

  • Gibson, K. B., Vo, D. T., & Nguyen, T. Q. (2012). An investigation of dehazing effects on image and video coding. IEEE Transactions on Image Processing, 21(2), 662–673.

    Article  MathSciNet  Google Scholar 

  • He, K., Sun, J., & Tang, X. (2009). Single image haze removal using dark channel prior. In IEEE conference on computer vision and pattern recognition, (pp. 1956–1963).

  • He, K., Sun, J., & Tang, X. (2011). Single image haze removal using dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(12), 2341–2353.

    Article  Google Scholar 

  • He, K., Sun, J., & Tang, X. (2013). Guided image filtering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(6), 1397–1409.

    Article  Google Scholar 

  • Kopf, J., Neubert, B., Chen, B., Cohen, M., Cohen-Or, D., Deussen, O., et al. (2008). Deep photo: Model-based photograph enhancement and viewing. ACM Transactions on Graphics, 27(5), 116.

    Article  Google Scholar 

  • Koschmieder, H. (1924). Theorie der horizontalen sichtweite. Beitrage zur Physik der freien Atmosphare (pp. 33–53).

  • Kratz, L., & Nishino, K. (2009). Factorizing scene albedo and depth from a single foggy image. In IEEE international conference on computer vision, (pp. 1701–1708).

  • Li, B., Peng, X., Wang, Z., Xu, J., & Feng, D. (2017). Aod-net: All-in-one dehazing network. In IEEE international conference on computer vision.

  • Li, B., Ren, W., Fu, D., Tao, D., Feng, D., Zeng, W., et al. (2018). Benchmarking single-image dehazing and beyond. IEEE Transactions on Image Processing, 28(1), 492–505.

    Article  MathSciNet  Google Scholar 

  • Li, S., Araujo, I. B., Ren, W., Wang, Z., Tokuda, E. K., Junior, R. H., Cesar-Junior, R., Zhang, J., Guo, X., & Cao, X. (2019). Single image deraining: A comprehensive benchmark analysis. In IEEE conference on computer vision and pattern recognition, (pp. 3838–3847).

  • Li, Y., Tan, R. T., & Brown, M. S. (2015). Nighttime haze removal with glow and multiple light colors. In IEEE international conference on computer vision, (pp. 226–234).

  • Li, Z., Tan, P., Tan, R. T., Zou, D., Zhou, S. Z., & Cheong, L. F. (2015). Simultaneous video defogging and stereo reconstruction. In IEEE conference on computer vision and pattern recognition, (pp. 4988–4997).

  • Liu, Y., Zhao, G., Gong, B., Li, Y., Raj, R., Goel, N., Kesav, S., Gottimukkala, S., Wang, Z., Ren, W., et al. (2018). Improved techniques for learning to dehaze and beyond: A collective study. arXiv preprint arXiv:1807.00202

  • Meng, G., Wang, Y., Duan, J., Xiang, S., & Pan, C. (2013). Efficient image dehazing with boundary constraint and contextual regularization. In IEEE international conference on computer vision, (pp. 617–624).

  • Nair, V., & Hinton, G. E. (2010). Rectified linear units improve restricted Boltzmann machines. In International conference on machine learning, (pp. 807–814).

  • Narasimhan, S. G., & Nayar, S. K. (2000). Chromatic framework for vision in bad weather. In IEEE conference on computer vision and pattern recognition, (pp. 598–605).

  • Narasimhan, S. G., & Nayar, S. K. (2003). Contrast restoration of weather degraded images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(6), 713–724.

    Article  Google Scholar 

  • Nayar, S. K., & Narasimhan, S. G. (1999). Vision in bad weather. In IEEE international conference on computer vision, (pp. 820–827).

  • Nishino, K., Kratz, L., & Lombardi, S. (2012). Bayesian defogging. International Journal of Computer Vision, 98(3), 263–278.

    Article  MathSciNet  Google Scholar 

  • Ren, W., & Cao, X. (2017). Deep video dehazing. In Pacific rim conference on multimedia, (pp. 14–24).

  • Ren, W., Liu, S., Zhang, H., Pan, J., Cao, X., & Yang, M. H. (2016). Single image dehazing via multi-scale convolutional neural networks. In European conference on computer vision, (pp. 154–169).

  • Ren, W., Ma, L., Zhang, J., Pan, J., Cao, X., Liu, W., & Yang, M. H. (2018). Gated fusion network for single image dehazing. In IEEE conference on computer vision and pattern recognition, (pp. 3253–3261).

  • Ren, W., Zhang, J., Xu, X., Ma, L., Cao, X., Meng, G., et al. (2018). Deep video dehazing with semantic segmentation. IEEE Transactions on Image Processing, 28(4), 1895–1908.

    Article  MathSciNet  Google Scholar 

  • Saxena, A., Sun, M., & Ng, A. Y. (2009). Make3d: Learning 3d scene structure from a single still image. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(5), 824–840.

    Article  Google Scholar 

  • Scharstein, D., & Szeliski, R. (2002). A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47(1–3), 7–42.

    Article  Google Scholar 

  • Scharstein, D., & Szeliski, R. (2003). High-accuracy stereo depth maps using structured light. In IEEE conference on computer vision and pattern recognition, (Vol. 1, pp. I–195).

  • Schaul, L., Fredembach, C., & Süsstrunk, S. (2009). Color image dehazing using the near-infrared. In IEEE international conference on image processing, (pp. 1629–1632).

  • Schechner, Y. Y., Narasimhan, S. G., & Nayar, S. K. (2001). Instant dehazing of images using polarization. In IEEE conference on computer vision and pattern recognition, (Vol. 1, pp. I–325).

  • Shwartz, S., Namer, E., & Schechner, Y. Y. (2006). Blind haze separation. In IEEE conference on computer vision and pattern recognition, (pp. 1984–1991).

  • Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012). Indoor segmentation and support inference from RGBD images. In European conference on computer vision, (pp. 746–760).

  • Sulami, M., Glatzer, I., Fattal, R., & Werman, M. (2014). Automatic recovery of the atmospheric light in hazy images. In IEEE international conference on computational photography.

  • Tan, R. T. (2008). Visibility in bad weather from a single image. In IEEE conference on computer vision and pattern recognition, (pp. 1–8).

  • Tan, R. T., Pettersson, N., & Petersson, L. (2007). Visibility enhancement for roads with foggy or hazy scenes. In IEEE intelligent vehicles symposium, (pp. 19–24).

  • Tang, K., Yang, J., & Wang, J. (2014). Investigating haze-relevant features in a learning framework for image dehazing. In IEEE conference on computer vision and pattern recognition, (pp. 2995–3002).

  • Tarel, J. P., & Hautiere, N. (2009). Fast visibility restoration from a single color or gray level image. In IEEE international conference on computer vision, (pp. 2201–2208).

  • Tarel, J. P., Hautière, N., Caraffa, L., Cord, A., Halmaoui, H., & Gruyer, D. (2012). Vision enhancement in homogeneous and heterogeneous fog. Intelligent Transportation Systems Magazine, 4(2), 6–20.

    Article  Google Scholar 

  • Treibitz, T., & Schechner, Y. Y. (2009). Polarization: Beneficial for visibility enhancement? In IEEE conference on computer vision and pattern recognition, (pp. 525–532).

  • Wang, Y. K., & Fan, C. T. (2014). Single image defogging by multiscale depth fusion. IEEE Transactions on Image Processing, 23(11), 4826–4837.

    Article  MathSciNet  Google Scholar 

  • Xie, S., & Tu, Z. (2015). Holistically-nested edge detection. In IEEE international conference on computer vision, (pp. 1395–1403).

  • Zhang, H., & Patel, V. M. (2018). Densely connected pyramid dehazing network. In IEEE conference on computer vision and pattern recognition.

  • Zhang, H., Sindagi, V., & Patel, V. M. (2018). Multi-scale single image dehazing using perceptual pyramid deep network. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, (pp. 902–911).

  • Zhang, J., Cao, Y., & Wang, Z. (2014). Nighttime haze removal based on a new imaging model. In IEEE international conference on image processing, (pp. 4557–4561).

  • Zhang, S., He, F., Ren, W., & Yao, J. (2018). Joint learning of image detail and transmission map for single image dehazing. The Visual Computer, 34, 1–12.

    Article  Google Scholar 

  • Zhang, S., Ren, W., & Yao, J. (2018). Feed-net: Fully end-to-end dehazing. In IEEE international conference on multimedia and expo, (pp. 1–6).

  • Zhang, X. S., Gao, S. B., Li, C. Y., & Li, Y. J. (2015). A retina inspired model for enhancing visibility of hazy images. Frontiers in Computational Neuroscience, 9, 1–13.

  • Zhu, Q., Mai, J., & Shao, L. (2014). Single image dehazing using color attenuation prior. In British machine vision conference

  • Zhu, Q., Mai, J., & Shao, L. (2015). A fast single image haze removal algorithm using color attenuation prior. IEEE Transactions on Image Processing, 24(11), 3522–3533.

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

This work is supported by the National Key R&D Program of China (Grant No. 2018YFB0803701), Beijing Natural Science Foundation (No. KZ201910005007), National Natural Science Foundation of China (Nos. U1636214, U1803264, U1605252, 61802403, 61602464, 61872421, 61922043), Peng Cheng Laboratory Project of Guangdong Province PCL2018KP004, and the Natural Science Foundation of Jiangsu Province (No. BK20180471). The work of W. Ren is supported in part by the CCF-DiDi GAIA (YF20180101), CCF-Tencent Open Fund, Zhejiang Lab’s International Talent Fund for Young Professionals, and the Open Projects Program of the National Laboratory of Pattern Recognition. The work of M.-H. Yang is supported by Directorate for Computer and Information Science and Engineering (CAREER 1149783).

Author information

Authors and Affiliations

  1. Institute of Information Engineering, Chinese Academy of Sciences, Beijing, 100093, China

    Wenqi Ren, Hua Zhang & Xiaochun Cao

  2. School of Cyber Security, University of Chinese Academy of Sciences, Beijing, 100049, China

    Xiaochun Cao

  3. The Cyberspace Security Research Center, Peng Cheng Laboratory, Shenzhen, 518055, China

    Xiaochun Cao

  4. School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, 210094, China

    Jinshan Pan

  5. School of Engineering, University of California at Merced, Merced, CA, 95343, USA

    Ming-Hsuan Yang

  6. Tianjin University, Tianjin, 300072, China

    Wenqi Ren

Authors
  1. Wenqi Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jinshan Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hua Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaochun Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ming-Hsuan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaochun Cao.

Additional information

Communicated by Srinivasa Narasimhan.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Wenqi Ren and Jinshan Pan have contribute equally to this work.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ren, W., Pan, J., Zhang, H. et al. Single Image Dehazing via Multi-scale Convolutional Neural Networks with Holistic Edges. Int J Comput Vis 128, 240–259 (2020). https://doi.org/10.1007/s11263-019-01235-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-019-01235-8

Keywords

Search

Navigation