skip to main content
10.1145/3436369.3437419acmotherconferencesArticle/Chapter ViewAbstractPublication PagesiccprConference Proceedingsconference-collections
research-article

Stack-Based Scale-Recurrent Network for Image Deblurring

Published: 11 January 2021 Publication History

Abstract

Quite a few researches are devoted to eliminating motion blur using "coarse-to-fine" architecture. And it has shown its superiority in removing motion blur in some cases. However, there are still exiting problems: Complex network structure and large number of parameters, which makes the model difficult to train and result in expensive runtime. Poor quality and insufficient deblurring performance of images when simply using this architecture to remove motion blur. To solve the above problems, In this paper, we utilize the "coarse-to-fine" architecture and combine the benefits of stacked structure, proposing a new structure "Stack-based Scale-recurrent Network" (SSRN) for image deblurring task. Different from other traditional multi-scale methods with multiple losses evaluation, we only have one loss evaluation, and Scale-recurrent network (SRN) is employed in our model since it has simpler network structure and the results recovered through it are better than those via other networks. We choose the LFW face dataset for testing in our experiment. Due to the particularity of face images, in this way, we can verify whether our model has good deblurring performance for blurred face images. On the other hand, it can also laterally reflect whether the model can be applied to face recognition task (make the blurred face image clear so that it can be used for face recognition, the face recognition method will be described in the following chapters). We evaluate our model by means of calculating PSNR/SSIM value and accuracy of face recognition on a large blurring dataset with complex motion. As we expected, compared with other deblurring methods, both the PSNR/SSIM value and face recognition accuracy of deblurred image obtained by our model have improved significantly, which confirms the superiority of our proposed model in image deblurring task. At the end of this article, we display some deblurred images to show the high deblurring performance of our model in deblurring task.

References

[1]
Sunghyun Cho and Seungyong Lee. Fast motion deblurring. ACM Transactions on graphics, 28(5):145:1-145:8, 2009. 2.
[2]
T. F. Chan and C.-K. Wong. Total variation blind deconvolution. IEEE Trans. on Image Processing, 7(3):370--375, 1998.
[3]
L. Xu and J. Jia. Two-phase kernel estimation for robust motion deblurring. In ECCV, pages 157--170. Springer, 2010.
[4]
A. Goldstein and R. Fattal. Blur-kernel estimation from spectral irregularities. In ECCV, pages 622--635. Springer, 2012.
[5]
J. Pan, Z. Hu, Z. Su, and M.-H. Yang. Deblurring text images via 10-regularized intensity and gradient prior. In CVPR, pages 2901--2908. IEEE, 2014.
[6]
J. Pan, D. Sun, H. Pfifister, and M.-H. Yang. Blind image deblurring using dark channel prior. In CVPR, pages 1628--1636. IEEE, 2016.
[7]
Q. Shan, J. Jia, and A. Agarwala. High-quality motion deblurring from a single image. volume 27, page 73. ACM, 2008.
[8]
A. Levin, Y. Weiss, F. Durand, and W. T. Freeman. Understanding and evaluating blind deconvolution algorithms. In CVPR, pages 1964--1971. IEEE, 2009.
[9]
S. Cho and S. Lee. Fast motion deblurring. In ACM Trans. on Graphics, volume 28, page 145. ACM, 2009.
[10]
D. Krishnan and R. Fergus. Fast image deconvolution using hyper-laplacian priors. In NIPS, pages 1033--1041, 2009.
[11]
L. Xu, S. Zheng, and J. Jia. Unnatural 10 sparse representation for natural image deblurring. In CVPR, pages 1107--1114. IEEE, 2013.
[12]
O. Whyte, J. Sivic, A. Zisserman, and J. Ponce. Non-uniform deblurring for shaken images. 2010.
[13]
A. Gupta, N. Joshi, C. L. Zitnick, M. Cohen, and B. Curless. Single image deblurring using motion density functions. In ECCV, pages 171--184. Springer, 2010.
[14]
M. Hirsch, C. J. Schuler, S. Harmeling, and B. Schölkopf. Fast removal of non-uniform camera shake. In ICCV, 2011.
[15]
S. Harmeling, H. Michael, and B. Schölkopf. Spacevariant single-image blind deconvolution for removing camera shake. In Advances in Neural Information Processing Systems, pages 829--837, 2010.
[16]
C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. ArXiv e-prints, Sept. 2016.
[17]
C. Dong, C. C. Loy, K. He, and X. Tang. Learning a deep convolutional network for image super-resolution. In ECCV, 2014. 2
[18]
Jun Yu, Min Tan, Hongyuan Zhang, Dacheng Tao, Yong Rui, Hierarchical Deep Click Feature Prediction for Fine-grained Image Recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence. 2019.
[19]
P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. arxiv, 2016.
[20]
Jun Yu, Zhenzhong Kuang, Baopeng Zhang, Wei Zhang, Dan Lin, Jianping Fan, Leveraging Content Sensitiveness and User Trustworthiness to Recommend Fine-Grained Privacy Settings for Social Image Sharing, IEEE Transactions on Information Forensics and Security, vol. 13, no. 5, pp. 1317--1332, 2018.
[21]
Y. Li, J.-B. Huang, N. Ahuja, and M.-H. Yang. Deep joint image fifiltering. In ECCV, 2016.
[22]
T. Hyun Kim, B. Ahn, and K. Mu Lee. Dynamic scene deblurring. In ICCV, pages 3160--3167. IEEE, 2013.
[23]
Jiawei Zhang, Jinshan Pan, Jimmy Ren, Yibing Song, Linchao Bao, Rynson WH Lau, and Ming-Hsuan Yang. Dynamic scene deblurring using spatially variant recurrent neural networks. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 2521--2529, 2018.
[24]
C. J. Schuler, M. Hirsch, S. Harmeling, and B. Scholkopf. Learning to deblur. TPAMI, 2016.
[25]
A. Chakrabarti. A neural approach to blind motion deblurring. In ECCV, 2016.
[26]
P. Svoboda, M. Hradis, L. Marˇˇ'sik, and P. Zemćik. Cnn for license plate motion deblurring. In ICIP, 2016.
[27]
L. Xu, J. S. Ren, C. Liu, and J. Jia. Deep convolutional neural network for image deconvolution. In NIPS, 2014.
[28]
D. Gong, J. Yang, L. Liu, Y. Zhang, I. Reid, C. Shen, A. v. d. Hengel, and Q. Shi. From motion blur to motion flflow: a deep learning solution for removing heterogeneous motion blur. In CVPR, 2017.
[29]
M. Noroozi, P. Chandramouli, and P. Favaro. Motion deblurring in the wild. In German Conference on Pattern Recognition, 2017.
[30]
J. Sun, W. Cao, Z. Xu, and J. Ponce. Learning a convolutional neural network for non-uniform motion blur removal. In CVPR, 2015.
[31]
Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 257--265, 2017.
[32]
Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. Scale-recurrent network for deep image deblurring. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 8174--8182, 2018.
[33]
Hongguang Zhang, Yuchao Dai, Hongdong Li, etc. Deep Stacked Hierarchical Multi-patch Network for Image Deblurring. In CVPR, pages 5979--5986, 2019.
[34]
Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiri Matas. Deblurgan: Blind motion deblurring using conditional adversarial networks. arXiv preprint arXiv: 1711.07064, 2017.
[35]
Jun Yu, Chaoyang Zhu, Jian Zhang, Qingming Huang, and Dacheng Tao, Spatial Pyramid-Enhanced NetVLAD with and Weighted Triplet Loss for Place Recognition, IEEE Transactions on Neural Networks and Learning Systems. 2019.
[36]
D. Eigen and R. Fergus. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In ICCV, pages 2650--2658, 2015.
[37]
M. Mathieu, C. Couprie, and Y. LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprint arXiv:1511.05440, 2015.
[38]
A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. van der Smagt, D. Cremers, and T. Brox. Flownet: Learning optical flflow with convolutional networks. In CVPR, pages 2758--2766, 2015.
[39]
Seungjun Nah Sanghyun Son Kyoung Mu Lee. Recurrent Neural Networks with Intra-Frame Iterations for Video Deblurring. In CVPR, pages 8103--8111, 2019.
[40]
Z. Liu, R. Yeh, X. Tang, Y. Liu, and A. Agarwala. Video frame synthesis using deep voxel flflow. In ICCV. IEEE, 2017.
[41]
S. Su, M. Delbracio, J. Wang, G. Sapiro, W. Heidrich, and O. Wang. Deep video deblurring. pages 1279--1288, 2017.
[42]
X. Tao, H. Gao, R. Liao, J. Wang, and J. Jia. Detail-revealing deep video super-resolution. In ICCV. IEEE, 2017.
[43]
N. Xu, B. Price, S. Cohen, and T. Huang. Deep image matting. In CVPR. IEEE, 2017.
[44]
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, pages 770--778. IEEE, 2016.
[45]
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2014.

Index Terms

  1. Stack-Based Scale-Recurrent Network for Image Deblurring

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    ICCPR '20: Proceedings of the 2020 9th International Conference on Computing and Pattern Recognition
    October 2020
    552 pages
    ISBN:9781450387835
    DOI:10.1145/3436369
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    In-Cooperation

    • Beijing University of Technology

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 11 January 2021

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Motion deblur
    2. coarse-to-fine architecture
    3. scale-recurrent network
    4. stacked structure

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Funding Sources

    • the National Natuarl Science Foundation of China
    • Distinguished Young Scientific Research Talents Plan in Universities of Fujian Province
    • the Program for New Century Excellent Talents in Universities of Fujian Province
    • the Fujian Provincial Natural Science Foundation of China

    Conference

    ICCPR 2020

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 53
      Total Downloads
    • Downloads (Last 12 months)5
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 15 Feb 2025

    Other Metrics

    Citations

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media