Skip to main content

Advertisement

Log in

DeblurExpandNet: image motion deblurring network aided by inertial sensor data

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

In this paper, we propose a convolutional neural network for the removal of spatially varying motion blur from captured images with the assistance of inertial sensor data. In the proposed system, both the image and motion data are captured simultaneously and passed to a network for processing. The proposed network adopts three parallel nets to extract image features and a per-pixel concatenation to tightly integrate motion homographies estimated from inertial sensor data with the input degraded image. This unique network design facilitates the use of homographies which describes the motion blur kernel more accurately. Compared to the recently proposed image deblurring networks, the proposed network is found to produce restored images that have fewer artifacts and provide quantifiable and subjective improvement.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Berger, M., Cole, M., Cole, M., Levy, S.: Geometry. No. v. 2 in Geometry I [-II]. Springer (1987). https://books.google.com/books?id=MoYZnQAACAAJ

  2. Faugeras, O.D., Lustman, F.: Motion and structure from motion in a piecewise planar environment. IJPRAI 2, 485–508 (1988)

    Google Scholar 

  3. Fergus, R., Singh, B., Hertzmann, A., Roweis, S.T., Freeman, W.T.: Removing camera shake from a single photograph. ACM Trans. Graph. 25(3), 787–794 (2006)

    Article  Google Scholar 

  4. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90

  5. Hu, Z., Yuan, L., Lin, S., Yang, M.: Image deblurring using smartphone inertial sensors. In: Proceedings of the 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp. 1855–1864 (2016). https://doi.org/10.1109/CVPR.2016.205

  6. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd international conference on international conference on machine learning, Vol 37, ICML’15, pp. 448–456. JMLR.org (2015)

  7. Jiao, J., Tu, W.C., He, S.W.H., Lau, R.: Formresnet: formatted residual learning for image restoration. In: Proceedings of the 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp. 1034–1042 (2017). https://doi.org/10.1109/CVPRW.2017.140

  8. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. CoRR abs/1412.6980 (2015)

  9. Klambauer, G., Unterthiner, T., Mayr, A., Hochreiter, S.: Self-normalizing neural networks. CoRR abs/1706.02515 (2017)

  10. Krishnan, D., Tay, T., Fergus, R.: Blind deconvolution using a normalized sparsity measure. In: CVPR 2011, pp. 233–240 (2011). https://doi.org/10.1109/CVPR.2011.5995521

  11. Kupyn, O., Budzan, V., Mykhailych, M., Mishkin, D., Matas, J.: DeblurGAN: blind motion deblurring using conditional adversarial networks. In: Proceedings of the 2018 IEEE/CVF conference on computer vision and pattern recognition, pp. 8183–8192 (2018)

  12. Kupyn, O., Martyniuk, T., Wu, J., Wang, Z.: Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better. In: The IEEE International Conference on Computer Vision (ICCV) (2019)

  13. Marnerides, D., Bashford-Rogers, T., Hatchett, J., Debattista, K.: Expandnet: a deep convolutional neural network for high dynamic range expansion from low dynamic range content. Comput. Graph. Forum 37(2), 37–49 (2018). https://doi.org/10.1111/cgf.13340

    Article  Google Scholar 

  14. Mustaniemi, J., Kannala, J., Särkkä, S., Matas, J., Heikkilä, J.: Gyroscope-aided motion deblurring with deep networks. In: Proceedings of the 2019 IEEE winter conference on applications of computer vision (WACV) pp. 1914–1922 (2018)

  15. Nah, S., Kim, T.H., Lee, K.M.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: Proceedings of the 2017 IEEE conference on computer vision and pattern recognition (CVPR) pp. 257–265 (2017)

  16. Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: J. Fürnkranz, T. Joachims (eds.) ICML, pp. 807–814. Omnipress (2010)

  17. Park, S.H., Levoy, M.: Gyro-based multi-image deconvolution for removing handshake blur. In: Proceedings of the 2014 IEEE conference on computer vision and pattern recognition, pp. 3366–3373 (2014). https://doi.org/10.1109/CVPR.2014.430

  18. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch (2017)

  19. Shan, Q., Jia, J., Agarwala, A.: High-quality motion deblurring from a single image. In: ACM SIGGRAPH 2008 Papers, SIGGRAPH ’08, pp. 73:1–73:10. ACM, New York, NY, USA (2008). http://doi.acm.org/10.1145/1399504.1360672

  20. Sim, H., Kim, M.: A deep motion deblurring network based on per-pixel adaptive kernels with residual down-up and up-down modules. In: Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2140–2149 (2019). https://doi.org/10.1109/CVPRW.2019.00267

  21. Sindelar, O., Sroubek, F.: Image deblurring in smartphone devices using built-in inertial measurement sensors. J. Elect. Imag. 22, 011003 (2013)

    Article  Google Scholar 

  22. Tai, Y., Tan, P., Brown, M.S.: Richardson-lucy deblurring for scenes under a projective motion path. IEEE Trans. Pattern Anal. Mach. Intell. 33(8), 1603–1618 (2011). https://doi.org/10.1109/TPAMI.2010.222

    Article  Google Scholar 

  23. Tao, X., Gao, H., Shen, X., Wang, J., Jia, J.: Scale-recurrent network for deep image deblurring. In: Proceedings of the 2018 IEEE conference on computer vision and pattern recognition, CVPR 2018, Salt Lake City, UT, USA, pp. 8174–8182 (2018). https://doi.org/10.1109/CVPR.2018.00853

  24. Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. In: International conference on learning representations (ICLR) (2016)

  25. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H., Shao, L.: Multi-stage progressive image restoration. In: CVPR (2021)

  26. Zhang, H., Dai, Y., Li, H., Koniusz, P.: Deep stacked hierarchical multi-patch network for image deblurring. In: The IEEE conference on computer vision and pattern recognition (CVPR) (2019)

  27. Zhang, S., Zhen, A., Stevenson, R.L.: Deep motion blur removal using noisy/blurry image pairs (2019)

  28. Zhang, S., Zhen, A., Stevenson, R.L.: A dataset for deep image deblurring aided by inertial sensor data. In: Fast track article for IS&T International Symposium on Electronic Imaging 2020: Computational Imaging XVIII proceedings. pp. 379–1–379–6(6) (2020). https://doi.org/10.2352/ISSN.2470-1173.2019.13.COIMG-136

  29. Zhen, R., Stevenson, R.: Inertial sensor aided multi-image nonuniform motion blur removal based on motion decomposition. J. Elect. Imag. 27(5), 053026 (2018)

    Google Scholar 

  30. Zhen, R., Stevenson, R.L.: Multi-image motion deblurring aided by inertial sensors. J. Elect. Imag. 25, 013027 (2016). https://doi.org/10.1117/1.JEI.25.1.013027

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Shuang Zhang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 2334 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, S., Zhen, A. & Stevenson, R.L. DeblurExpandNet: image motion deblurring network aided by inertial sensor data. SIViP 16, 1169–1176 (2022). https://doi.org/10.1007/s11760-021-02067-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-021-02067-1

Keywords