Abstract
Deblurring, denoising and super-resolution (SR) are important image recovery tasks that are committed to improving image quality. Despite the rapid development of deep learning and vast studies on improving image quality have been proposed, the most existing recovery solutions simply deal with quality degradation caused by a single distortion factor, such as SR focusing on improving spatial resolution. Since very little work has been done to analyze the interaction and characteristics of the deblurring, denoising and SR mixing problems, this paper considers the multi-distortion image recovery problem from a holistic perspective and introduces an end-to-end multi-stream fusion network (MSFN) to restore a multi-distortion image (low-resolution image with noise and blur) into a clear high-resolution (HR) image. Firstly, MSFN adopts multiple reconstruction branches to extract deblurring, denoise and SR features with respect to different degradations. Then, MSFN gradually fuses these multi-stream recovery features in a determined order and obtains an enhanced restoration feature by using two fusion modules. In addition, MSFN uses fusion modules and residual attention modules to facilitate the fusion of different recovery features from the denoising branch and the deblurring branch for the trunk SR branch. Experiments on several benchmarks fully demonstrate the superiority of our MSFN in solving the multi-distortion image recovery problem.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Zhang, B., Sheng, B., Li, P., Lee, T.: Depth of field rendering using multilayer-neighborhood optimization. IEEE Trans. Vis. Comput. Graph. 26(8), 2546–2559 (2020)
Nazir, A., et al.: OFF-eNET: an optimally fused fully end-to-end network for automatic dense volumetric 3d intracranial blood vessels segmentation. IEEE Trans. Image Process. 29, 7192–7202 (2020)
Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: European Conference on Computer Vision, pp. 184–199 (2014)
Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 391–407. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_25
Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
Lai, W., Huang, J., Ahuja, N., Yang, M.: Deep laplacian pyramid networks for fast and accurate super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2017, pp. 5835–5843 (2017)
Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2017)
Chen, Y., Liu, S., Wang, X.: Learning continuous image representation with local implicit image function, CoRR, vol. abs/2012.09161 (2020)
Gu, J., Dong, C.: Interpreting super-resolution networks with local attribution maps, CoRR, vol. abs/2011.11036 (2020)
Xu, L., Ren, J.S., Liu, C., Jia, J.: Deep convolutional neural network for image deconvolution. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 27, Curran Associates Inc, pp. 1790–1798 (2014). http://papers.nips.cc/paper/5485-deep-convolutional-neural-network-for-image-deconvolution.pdf
Hradi\(\check{s}\), M.: Convolutional neural networks for direct text deblurring. In: British Machine Vision Conference 2015 (2015)
Matas, J.: Deblurgan: blind motion deblurring using conditional adversarial networks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
Xin, T., Gao, H., Shen, X., Wang, J., Jia, J.: Scale-recurrent network for deep image deblurring. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
Madam, N.T., Kumar, S., Rajagopalan, A.N.: Unsupervised class-specific deblurring. In: 15th European Conference, Munich, Germany, 8–14 September 2018, proceedings, part x (2018)
Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space. In: 2007 IEEE International Conference on Image Processing. IEEE (September 2007)
Gu, S., Zhang, L., Zuo, W., Feng, X.: Weighted nuclear norm minimization with application to image denoising, pp. 2862–2869 (2014)
Lefkimmiatis, S.: Non-local color image denoising with convolutional neural networks, pp. 5882–5891 (2017)
Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (2017)
Zhang, K., Zuo, W., Zhang, L.: Ffdnet: toward a fast and flexible solution for CNN-based image denoising. IEEE Trans. Image Process. 27(9), 4608–4622 (2018)
Tai, Y., Yang, J., Liu, X., Xu, C.: Memnet: a persistent memory network for image restoration, pp. 4549–4557 (2017)
Guo, S., Yan, Z., Zhang, K., Zuo, W., Zhang, L.: Toward convolutional blind denoising of real photographs, pp. 1712–1722 (2019)
Zhang, X., Dong, H., Hu, Z., Lai, W.-S., Wang, F., Yang, M.-H.: Gated fusion network for joint image deblurring and super-resolution, CoRR, vol. abs/1807.10806 (2018)
Liang, Z., Zhang, D., Shao, J.: Jointly solving deblurring and super-resolution problems with dual supervised network. In: IEEE International Conference on Multimedia and Expo (ICME), vol. 2019, pp. 790–795 (2019)
Anwar, S., Barnes, N.: Real image denoising with feature attention. In: IEEE International Conference on Computer Vision, pp. 3155–3164 (2019)
Nah, S., Kim, T.H., Lee, K.M.: Deep multi-scale convolutional neural network for dynamic scene deblurring, pp. 257–265 (2017)
Acknowledgement
This work was supported in part by the National Natural Science Foundation of China under Grants 62077037 and 61872241, in part by Shanghai Municipal Science and Technology Major Project under Grant 2021SHZDZX0102, in part by the Science and Technology Commission of Shanghai Municipality under Grants 18410750700 and 17411952600, in part by Shanghai Lin-Gang Area Smart Manufacturing Special Project under Grant ZN2018020202-3, and in part by Project of Shanghai Municipal Health Commission(2018ZHYL0230).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Wen, Y. et al. (2021). Multi-Stream Fusion Network for Multi-Distortion Image Super-Resolution. In: Magnenat-Thalmann, N., et al. Advances in Computer Graphics. CGI 2021. Lecture Notes in Computer Science(), vol 13002. Springer, Cham. https://doi.org/10.1007/978-3-030-89029-2_19
Download citation
DOI: https://doi.org/10.1007/978-3-030-89029-2_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-89028-5
Online ISBN: 978-3-030-89029-2
eBook Packages: Computer ScienceComputer Science (R0)