Bingcai Wei
ABSTRACT
Convolutional neural network (CNN) is widely used in the field of image restoration, however, most existing CNN based image restoration methods only focus on a part of image restoration without considering the relationship between image deblurring, image de-raining and image denoising. In addition, training a neural network model for different image restoration tasks requires a large amount of training data, plenty of hardware overheads and a great deal of time. In order to reduce resource consumption and improve the generality of the model, this paper proposes an image restoration algorithm using multistage progressive encoder-decoder network with attention and transfer learning (MSP-ATL). First of all, we design a multi-stage progressive encoder-decoder network with attention mechanisms, which does not require down-sampling or fragmentation of the input, and can retain the overall information of the image. Secondly, following the idea of coarse-to-fine which is widely used in the field of image restoration, we propose a multistage progressive loss function to recover images from coarse to fine by cooperating with the multi-stage network structure. Finally, using transfer learning, the obtained deblurring model can be transferred to image de-raining and image denoising tasks with less training data and simple training process. Extensive experimental results on commonly used datasets demonstrate the efficiency and effectiveness of the proposed method.
- Sun, J., Cao, W., Xu, Z., & Ponce, J. , 2015. Learning a convolutional neural network for non-uniform motion blur removal. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 769-777). DOI: https://doi.org/10.1109/CVPR.2015.7298677Google Scholar
Cross Ref
- Nah, S., Hyun Kim, T., & Mu Lee, K. , 2017. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3883-3891). DOI: https://doi.org/10.1109/CVPR.2017.35Google ScholarCross Ref
- Tao, X., Gao, H., Shen, X., Wang, J., & Jia, J. , 2018. Scale-recurrent network for deep image deblurring. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8174-8182). DOI: https://doi.org/10.1109/CVPR.2018.00853Google ScholarCross Ref
- Kupyn, O., Budzan, V., Mykhailych, M., Mishkin, D., & Matas, J. , 2018. Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8183-8192). DOI: https://doi.org/10.1109/CVPR.2018.00854Google ScholarCross Ref
- Kupyn, O., Martyniuk, T., Wu, J., & Wang, Z. , 2019. Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 8878-8887). DOI: https://doi.org/10.1109/ICCV.2019.00897Google ScholarCross Ref
- Zhang, K., Luo, W., Zhong, Y., Ma, L., Stenger, B., Liu, W., & Li, H. , 2020. Deblurring by realistic blurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2737-2746). DOI: https://doi.org/10.1109/CVPR42600.2020.00281Google ScholarCross Ref
- Zheng, X., Liao, Y., Guo, W., Fu, X., & Ding, X. , 2013, November). Single-image-based rain and snow removal using multi-guided filter. In International conference on neural information processing (pp. 258-265). Springer, Berlin, Heidelberg. DOI: https://doi.org/10.1007/978-3-642-42051-1_33Google ScholarCross Ref
- Yang, W., Tan, R. T., Feng, J., Liu, J., Guo, Z., & Yan, S. , 2017. Deep joint rain detection and removal from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1357-1366). DOI: https://doi.org/10.1109/CVPR.2017.183Google ScholarCross Ref
- Zhang, H., Sindagi, V., & Patel, V. M. , 2019. Image de-raining using a conditional generative adversarial network. IEEE transactions on circuits and systems for video technology, 30(11), 3943-3956. DOI: https://doi.org/ 10.1109/TCSVT.2019.2920407Google Scholar
- Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., & Paisley, J. , 2017. Removing rain from single images via a deep detail network. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3855-3863). DOI: https://doi.org/10.1109/CVPR.2017.186Google ScholarCross Ref
- Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. , 2007. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Transactions on image processing, 16(8), 2080-2095. DOI: https://doi.org/ 10.1109/TIP.2007.901238Google ScholarCross Ref
- Dong, W., Zhang, L., Shi, G., & Li, X. , 2012. Nonlocally centralized sparse representation for image restoration. IEEE transactions on Image Processing, 22(4), 1620-1630. DOI: https://doi.org/10.1109/TIP.2012.2235847Google ScholarDigital Library
- Gu, S., Zhang, L., Zuo, W., & Feng, X. , 2014. Weighted nuclear norm minimization with application to image denoising. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2862-2869). DOI: https://doi.org/10.1109/CVPR.2014.366Google ScholarDigital Library
- Xu, J., Zhang, L., Zhang, D., & Feng, X. , 2017. Multi-channel weighted nuclear norm minimization for real color image denoising. In Proceedings of the IEEE international conference on computer vision (pp. 1096-1104). DOI: https://doi.org/10.1109/ICCV.2017.125Google ScholarCross Ref
- Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, L. , 2017. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing, 26(7), 3142-3155. DOI: https://doi.org/10.1109/TIP.2017.2662206Google ScholarDigital Library
- He, K., Zhang, X., Ren, S., & Sun, J. , 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778). DOI: https://doi.org/10.1109/CVPR.2016.90Google ScholarCross Ref
- Yu, F., & Koltun, V. , 2015. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122. DOI: https://doi.org/10.48550/arXiv.1511.07122Google Scholar
- Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. , 2017. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4700-4708). DOI: https://doi.org/10.1109/CVPR.2017.243Google ScholarCross Ref
- Mao, X., Shen, C., & Yang, Y. B. , 2016. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. Advances in neural information processing systems, 29. DOI: https://doi.org/10.48550/arXiv.1603.09056Google Scholar
- Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., ... & Bengio, Y. , 2020. Generative adversarial networks. Communications of the ACM, 63(11), 139-144. DOI: https://doi.org/10.1145/3422622Google ScholarDigital Library
- Zamir, S. W., Arora, A., Khan, S., Hayat, M., Khan, F. S., Yang, M. H., & Shao, L. , 2021. Multi-stage progressive image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 14821-14831). DOI: https://doi.org/10.1109/CVPR46437.2021.01458Google ScholarCross Ref
- Zhang, H., Dai, Y., Li, H., & Koniusz, P. , 2019. Deep stacked hierarchical multi-patch network for image deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5978-5986). DOI: https://doi.org/10.1109/CVPR.2019.00613Google ScholarCross Ref
- Geraci, A. , 1991. IEEE standard computer dictionary: Compilation of IEEE standard computer glossaries. IEEE Press.Google ScholarDigital Library
- Huang, D. A., Kang, L. W., Yang, M. C., Lin, C. W., & Wang, Y. C. F. , 2012, July Context-aware single image rain removal. In 2012 IEEE International Conference on Multimedia and Expo (pp. 164-169). IEEE. DOI: https://doi.org/10.1109/ICME.2012.92Google ScholarDigital Library
- Woo, S., Park, J., Lee, J. Y., & Kweon, I. S. , 2018. Cbam: Convolutional block attention module. In Proceedings of the European conference on computer vision (ECCV) (pp. 3-19). DOI: https://doi.org/10.1007/978-3-030-01234-2_1Google ScholarDigital Library
- Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. , 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. DOI: https://doi.org/10.48550/arXiv.1406.1078Google Scholar
- Chaudhari, S., Mithal, V., Polatkan, G., & Ramanath, R. , 2021. An attentive survey of attention models. ACM Transactions on Intelligent Systems and Technology (TIST), 12(5), 1-32. DOI: https://doi.org/10.1145/3465055Google ScholarDigital Library
- Zagoruyko, S., & Komodakis, N. , 2016. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928. DOI: https://doi.org/10.48550/arXiv.1612.03928Google Scholar
- Wang, Y. Q. , 2014, October. A multilayer neural network for image demosaicking. In 2014 IEEE International Conference on Image Processing (ICIP) (pp. 1852-1856). IEEE. DOI: https://doi.org/ 10.1109/ICIP.2014.7025371Google Scholar
- Wen, Y., Chen, J., Sheng, B., Chen, Z., Li, P., Tan, P., & Lee, T. Y. , 2021. Structure-aware motion deblurring using multi-adversarial optimized cyclegan. IEEE Transactions on Image Processing, 30, 6142-6155. DOI: https://doi.org/10.1109/TIP.2021.3092814Google ScholarCross Ref
- Xu, L., Zheng, S., & Jia, J. , 2013. Unnatural l0 sparse representation for natural image deblurring. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1107-1114). DOI: https://doi.org/10.1109/CVPR.2013.147Google ScholarDigital Library
- Hyun Kim, T., Ahn, B., & Mu Lee, K. , 2013. Dynamic scene deblurring. In Proceedings of the IEEE international conference on computer vision (pp. 3160-3167). DOI: https://doi.org/10.1109/ICCV.2013.392Google ScholarDigital Library
- Whyte, O., Sivic, J., Zisserman, A., & Ponce, J. , 2012. Non-uniform deblurring for shaken images. International journal of computer vision, 98(2), 168-186. DOI: https://doi.org/10.1007/s11263-011-0502-7Google ScholarDigital Library
- Gong, D., Yang, J., Liu, L., Zhang, Y., Reid, I., Shen, C., ... & Shi, Q. , 2017. From motion blur to motion flow: A deep learning solution for removing heterogeneous motion blur. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2319-2328). DOI: https://doi.org/10.1109/CVPR.2017.405Google ScholarCross Ref
- Zhang, J., Pan, J., Ren, J., Song, Y., Bao, L., Lau, R. W., & Yang, M. H. , 2018. Dynamic scene deblurring using spatially variant recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2521-2529). DOI: https://doi.org/10.1109/CVPR.2018.00267Google ScholarCross Ref
- Shen, Z., Wang, W., Lu, X., Shen, J., Ling, H., Xu, T., & Shao, L. , 2019. Human-aware motion deblurring. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 5572-5581). DOI: https://doi.org/10.1109/ICCV.2019.00567Google ScholarCross Ref
- Gao, H., Tao, X., Shen, X., & Jia, J. , 2019. Dynamic scene deblurring with parameter selective sharing and nested skip connections. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 3848-3856). DOI: https://doi.org/10.1109/CVPR.2019.00397Google ScholarCross Ref
- Park, D., Kang, D. U., Kim, J., & Chun, S. Y. , 2020, August. Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training. In European Conference on Computer Vision (pp. 327-343). Springer, Cham. DOI: https://doi.org/10.1007/978-3-030-58539-6_20Google ScholarDigital Library
- Li, Y., Tan, R. T., Guo, X., Lu, J., & Brown, M. S. , 2016. Rain streak removal using layer priors. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2736-2744). DOI: https://doi.org/ 10.1109/CVPR.2016.299Google ScholarCross Ref
- Zhang, H., & Patel, V. M. , 2018. Density-aware single image de-raining using a multi-stream dense network. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 695-704). DOI: https://doi.org/ 10.1109/CVPR.2018.00079Google ScholarCross Ref
- Fu, X., Huang, J., Ding, X., Liao, Y., & Paisley, J. , 2017. Clearing the skies: A deep network architecture for single-image rain removal. IEEE Transactions on Image Processing, 26(6), 2944-2956. DOI: https://doi.org/ 10.1109/TIP.2017.2691802Google ScholarDigital Library
- Wei, W., Meng, D., Zhao, Q., Xu, Z., & Wu, Y. , 2019. Semi-supervised transfer learning for image rain removal. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 3877-3886). DOI: https://doi.org/10.1109/CVPR.2019.00400Google ScholarCross Ref
- Yasarla, R., & Patel, V. M. , 2019. Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8405-8414). DOI: https://doi.org/10.1109/CVPR.2019.00860Google ScholarCross Ref
- Li, X., Wu, J., Lin, Z., Liu, H., & Zha, H. , 2018. Recurrent squeeze-and-excitation context aggregation net for single image deraining. In Proceedings of the European conference on computer vision (ECCV) (pp. 254-269). DOI: https://doi.org/10.1007/978-3-030-01234-2_16Google ScholarDigital Library
- Ren, D., Zuo, W., Hu, Q., Zhu, P., & Meng, D. , 2019. Progressive image deraining networks: A better and simpler baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3937-3946). DOI: https://doi.org/10.1109/CVPR.2019.00406Google ScholarCross Ref
- Jiang, K., Wang, Z., Yi, P., Chen, C., Huang, B., Luo, Y., ... & Jiang, J. , 2020. Multi-scale progressive fusion network for single image deraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp.8346-8355). DOI: https://doi.org/10.1109/CVPR42600.2020.00837Google ScholarCross Ref
- Abdelhamed, A., Lin, S., & Brown, M. S. , 2018. A high-quality denoising dataset for smartphone cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1692-1700). DOI: https://doi.org/10.1109/CVPR.2018.00182Google ScholarCross Ref
- Burger, H. C., Schuler, C. J., & Harmeling, S. , 2012, June. Image denoising: Can plain neural networks compete with BM3D?. In 2012 IEEE conference on computer vision and pattern recognition (pp. 2392-2399). IEEE. DOI: https://doi.org/10.1109/CVPR.2012.6247952Google ScholarCross Ref
- Guo, S., Yan, Z., Zhang, K., Zuo, W., & Zhang, L. , 2019. Toward convolutional blind denoising of real photographs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1712-1722). DOI: https://doi.org/10.1109/CVPR.2019.00181Google ScholarCross Ref
- Roth, S., & Black, M. J. , 2009. Fields of experts. International Journal of Computer Vision, 82(2), 205-229. DOI: https://doi.org/10.1007/s11263-008-0197-6Google ScholarDigital Library
- Lebrun, M., Colom, M., & Morel, J. M. , 2015. The noise clinic: a blind image denoising algorithm. Image Processing On Line, 5, 1-54. DOI: https://doi.org/10.5201/ipol.2015.125Google ScholarCross Ref
- Aharon, M., Elad, M., & Bruckstein, A. , 2006. K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on signal processing, 54(11), 4311-4322. DOI: https://doi.org/ 10.1109/TSP.2006.881199Google ScholarDigital Library
Index Terms
- Image Restoration Using Multi-Stage Progressive Encoder-Decoder Network With Attention and Transfer Learning (MSP-ATL)
Recommendations
Spatially variant defocus blur map estimation and deblurring from a single image
A blur map estimation method using edge information is proposed.The blur map is segmented into multiple superpixels according to the image contours.Ringing artifacts and noise are detected and removed after deconvolution. In this paper, we propose a ...
DC-Deblur: A Dilated Convolutional Network for Single Image Deblurring
Intelligent Data Engineering and Automated Learning – IDEAL 2021AbstractSingle image deblurring is a significant and challenging task in image processing vision and machine learning. Convolutional Neural Network (CNN) based models for deblurring often have a complex structure and a considerable number of parameters ...
A survey of deep learning approaches to image restoration
AbstractIn this paper, we present an extensive review on deep learning methods for image restoration tasks. Deep learning techniques, led by convolutional neural networks, have received a great deal of attention in almost all areas of image ...
Comments