Abstract
Image quality is relevant to the performance of computer vision applications. The interference of rain streaks often greatly depreciates the visual effect of images. It is a traditional and critical vision challenge to remove rain streaks from rainy images. In this paper, we introduce a deep connectionist screen blend model for single-image rain removal research. The novel deep structure is mainly composed of shortcut connections, and ends with sibling branches. The specific architecture is designed for joint optimization of heterogeneous but related tasks. In particular, a feature-level task is design to preserve object edges which tend to be lost in de-rained images. Moreover, a comprehensive image quality assessment is an additional vision task for further improvement on de-rained results. Instead of using rules of thumb, we propose an actionable method to dynamically assign appropriate weighting coefficients for all vision tasks we use. On the other hand, various factors such as haze also give rise to weak visual appeal of rainy images. To remove these adverse factors, we develop an image enhancement framework which enables the hyperparameters to be optimized in an adaptive way, and efficiently improves the perceived quality of de-rained results. The effectiveness of the proposed de-raining system has been verified by extensive experiments, and most results of our method are impressive. The source code and more de-rained results will be available online.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Cai B, Xu X, Jia K, Qing C, Tao D (2016) Dehazenet: an end-to-end system for single image haze removal. IEEE Trans Image Process 25(11):5187–5198
Charbonnier P, Blancferaud L, Aubert G, Barlaud M (1997) Deterministic edge-preserving regularization in computed imaging. IEEE Trans Image Process 6(2):298–311
Chen J-C, Liu C-F (2017) Deep net architectures for visual-based clothing image recognition on large database. Soft Comput 21(11):2923–2939
Chen Y, Hsu C (2013) A generalized low-rank appearance model for spatio-temporally correlated rain streaks, pp 1968–1975
Deng J, Dong W, Socher R, Li L, Li K, Feifei L (2009) Imagenet: a large-scale hierarchical image database, pp 248–255
Fu X, Huang J, Zeng D, Huang Y, Ding X, Paisley J (2017) Removing rain from single images via a deep detail network, pp 3855–3863
Fu X, Liang B, Huang Y, Ding X, Paisley J (2019) Lightweight pyramid networks for image deraining. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2019.2926481
Gao Y, Hu H-M, Li B, Guo Q, Pu S (2019) Detail preserved single image dehazing algorithm based on airlight refinement. IEEE Trans Multimed 21(2):351–362
Girshick R (2015) Fast r-cnn
Goodfellow IJ, Pougetabadie J, Mirza M, Xu B, Wardefarley D, Ozair S, Courville AC, Bengio Y (2014) Generative adversarial nets, pp 2672–2680
He K, Sun J, Tang X (2011) Single image haze removal using dark channel prior. IEEE Trans Pattern Anal Mach Intell 33(12):2341–2353
He K, Gkioxari G, Dollar P, Girshick RB (2017) Mask r-cnn. In: International conference on computer vision, pp 2980–2988
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. Comput Vis Pattern Recognit. https://doi.org/10.1109/CVPR.2016.90
Jia Y, Jie L, Xin F (2018) No-reference quality assessment of contrast-distorted images using contrast enhancement
Johnson J, Alahi A, Feifei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: European conference on computer vision, pp 694–711
Kang LW, Lin CW, Fu YH (2012) Automatic single image based rain streaks removal via image decomposition. IEEE Trans Image Process 21(4):1742–1755
Khmag A, Ramli AR, Kamarudin N (2019) Clustering-based natural image denoising using dictionary learning approach in wavelet domain. Soft Comput 23:8013–8027. https://doi.org/10.1007/s00500-018-3438-9
Kim J-H, Lee C, Sim J-Y, Kim C-S (2013) Single-image deraining using an adaptive nonlocal means filter, pp 914–917
Kim JH, Sim JY, Kim C (2015) Video deraining and desnowing using temporal correlation and low-rank matrix completion. IEEE Trans Image Process 24(9):2658–2670
Kirkpatrick J, Pascanu R, Rabinowitz NC, Veness J, Desjardins G, Rusu AA, Milan K, Quan J, Ramalho T, Grabskabarwinska A et al (2017) Overcoming catastrophic forgetting in neural networks. Proc Natl Acad Sci USA 114(13):3521–3526
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105
Li Y, Tan RT, Guo X, Lu J, Brown MS (2016) Rain streak removal using layer priors, pp 2736–2744
Li X, Wu J, Lin Z, Liu H, Zha H (2018b) Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: European conference on computer vision. Springer, pp 262–277
Li M, Xie Q, Zhao Q, Wei W, Gu S, Tao J, Meng D (2018a) Video rain streak removal by multiscale convolutional sparse coding, pp 6644–6653
Luo Y, Xu Y, Ji H (2015) Removing rain from a single image via discriminative sparse coding, pp 3397–3405
Martin DR, Fowlkes CC, Tal D, Malik J (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. vol 2, pp 416–423
Matsui T, Ikehara M (2020) Gan-based rain noise removal from single-image considering rain composite models. IEEE Access 8:40892–40900
McCartney EJ (1976) Optics of the atmosphere: scattering by molecules and particles. Wiley, New York
Nair V, Hinton GE (2010) Rectified linear units improve restricted Boltzmann machines, pp 807–814
Narasimhan SG, Nayar SK (2003) Contrast restoration of weather degraded images. IEEE Trans Pattern Anal Mach Intell 25(6):713–724
Qu B, Suganthan PN, Liang J (2012) Differential evolution with neighborhood mutation for multimodal optimization. IEEE Trans Evol Comput 16(5):601–614
Rudin LI, Osher S, Fatemi E (1992) Nonlinear total variation based noise removal algorithms. Physica D 60(1–4):259–268
Santhaseelan V, Asari VK (2015) Utilizing local phase information to remove rain from video. Int J Comput Vis 112(1):71–89
Sheikh HR, Bovik AC (2004) Image information and visual quality. In: 2004 IEEE international conference on acoustics, speech, and signal processing, vol 3(iii), p 709
Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International conference on learning representations
Wang W, Yuan X (2017) Recent advances in image dehazing. IEEE/CAA J Autom Sinica 4(3):410–436
Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612
Xiang P, Wang L, Wu F, Cheng J, Zhou M (2019) Single-image de-raining with feature-supervised generative adversarial network. IEEE Signal Process Lett 26(5):650–654
Yang W, Tan RT, Feng J, Liu J, Yan S, Guo Z (2019) Joint rain detection and removal from a single image with contextualized deep networks. IEEE Trans Pattern Anal Mach Intell 42:1377–1393
Yang X, Li H, Fan Y, Chen R (2019) Single image haze removal via region detection network. IEEE Trans Multimed 21(10):2545–2560
Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. Comput Vis Pattern Recognit. https://doi.org/10.1109/CVPR.2017.183
Yin X, Liu X (2018) Multi-task convolutional neural network for pose-invariant face recognition. IEEE Trans Image Process 27(2):964–975
You S, Tan RT, Kawakami R, Mukaigawa Y, Ikeuchi K (2015) Adherent raindrop modeling, detection and removal in video. IEEE Trans Pattern Anal Mach Intell 38:1721–1733
Zhang X, Xiong Y (2009) Impulse noise removal using directional difference based noise detector and adaptive weighted mean filter. IEEE Signal Process Lett 16(4):295–298
Zhang H, Patel VM (2018) Density-aware single image de-raining using a multi-stream dense network. Comput Vis Pattern Recognit. https://doi.org/10.1109/CVPR.2018.00079
Zhao H, Xiao C, Yu J, Xu X (2015) Single image fog removal based on local extrema. IEEE/CAA J Autom Sinica 2(2):158–165
Acknowledgements
This work is supported by the National Natural Science Foundation of China (No. 61672122, No. 61402070, No. 61602077), the Natural Science Foundation of Liaoning Province of China (No. 20170540097, No. 2015020023), and the Fundamental Research Funds for the Central Universities (No. 3132016348), Next-Generation Internet Innovation Project of CERNET (No. NGII20181205).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Yulong Fan declares that he has no conflict of interest. Rong Chen declares that he has no conflict of interest. Yang Li declares that he has no conflict of interest. Tianlun Zhang declares that he has no conflict of interest.
Ethical approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Additional information
Communicated by V. Loia.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This work is supported by the National Natural Science Foundation of China under Grant 61672122, Grant 61602077, Grant 61772344 and Grant 61732011, the Public Welfare Funds for Scientific Research of Liaoning Province of China under Grant 20170005, the Natural Science Foundation of Liaoning Province of China under Grant 20170540097, and the Fundamental Research Funds for the Central Universities under Grant 3132016348.
Rights and permissions
About this article
Cite this article
Fan, Y., Chen, R., Li, Y. et al. Deep neural de-raining model based on dynamic fusion of multiple vision tasks. Soft Comput 25, 2221–2235 (2021). https://doi.org/10.1007/s00500-020-05291-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-020-05291-y