Skip to main content
SpringerLink
Log in

Image compressed sensing using multi-scale residual generative adversarial network

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Although faster and deeper convolutional networks have made breakthroughs in image compressed sensing (CS), there is still one central unsolved problem: how do we make the reconstructed image have more delicate texture details? The existing image CS algorithms are based on pixel loss to reconstruct the original image, which leads to the reconstructed image smoothness and lack of structural information. In order to solve the problem, this paper proposes MR-CSGAN: a multi-scale residual generative adversarial network (GAN) for image CS. MR-CSGAN combines multi-scale residual blocks by consisting of three different convolution kernels to exploit the image features fully. Furthermore, the perceptual loss is used as the objective optimization function instead of pixel loss to reconstruct a finer image. Experimental results show that the proposed MR-CSGAN can make the reconstructed image obtain more robust structural information and better visual effects than other state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Availability of data and material

Not applicable.

Code availability

(https://github.com/wen-jie-yuan/MR-CSGAN).

References

  1. Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory. 52(4), 1289–1306 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  2. Candès, E.J.: Compressive sampling. In: Proceedings of the International Congress of Mathematicians, pp. 1433–1452 (2006)

  3. Yuan, X., Brady, D.J., Katsaggelos, A.K.: Snapshot compressive imaging: theory, algorithms, and applications. IEEE Trans. Image Process. 38(2), 65–88 (2021)

    Google Scholar 

  4. Duarte, M.F., Davenport, M.A., Takhar, D., Laska, J.N., Sun, T., Kelly, K.F., Baraniuk, R.G.: Single-pixel imaging via compressive sampling. IEEE Signal Process. Mag. 25(2), 83–91 (2008)

    Article  Google Scholar 

  5. Liu, Y., Wu, S., Huang, X., Chen, B., Zhu, C.: Hybrid CS-DMRI: periodic time-variant subsampling and omnidirectional total variation based reconstruction. IEEE Trans. Med. Imaging. 36(10), 2148–2159 (2017). https://doi.org/10.1109/TMI.2017.2717502

    Article  Google Scholar 

  6. Zha, Z., Liu, X., Zhang, X.: Compressed sensing image reconstruction via adaptive sparse nonlocal regularization. Vis Comput. 34, 117–137 (2018). https://doi.org/10.1007/s00371-016-1318-9

    Article  Google Scholar 

  7. Li, C., Yin, W., Zhang, Y.: User’s guide for TVAL3: TV minimization by augmented lagrangian and alternating direction algorithms. CAAM Report 20, 46–47 (2009)

    Google Scholar 

  8. Chen, C., Tramel, E. W., Fowler, J. E.: Compressed-sensing recovery of images and video using multihypothesis predictions. In: Signals, Systems and Computers (ASILOMAR), pp. 1193-1198 (2011)

  9. Zhang, J., Zhao, D.: Group-based sparse representation for image restoration. IEEE Trans. Image Process. 23, 3336–3351 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  10. Metzler, C.A., Maleki, A., Baraniuk, R.G.: From denoising to compressed sensing. IEEE Trans. Inf. Theory. 62, 5117–5144 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  11. Zha, Z., Yuan, X., Wen, B., Zhou, J., Zhu, C.: Group sparsity residual constraint with non-local priors for image restoration. IEEE Trans. Image Process. 29, 8960–8975 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  12. Zha, Z., Wen, B., Yuan, X., Zhou, J., Zhu, C.: A hybrid structural sparsification error model for image restoration. In: IEEE Transactions on Neural Networks and Learning Systems (2021)

  13. Kulkarni, K., Lohit, S., Turaga, P., Kerviche, R., Ashok, A.: Reconnet: Non-iterative reconstruction of images from compressively sensed measurements. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 449–458 (2016)

  14. Zhang, J., Ghanem, B.: ISTA-Net: Interpretable optimization-inspired deep network for image compressive sensing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1828–1837 (2018)

  15. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J Imaging Sci. 2(1), 183–202 (2009). https://doi.org/10.1137/080716542

    Article  MathSciNet  MATH  Google Scholar 

  16. Shi, W., Jiang, F., Liu, S., Zhao, D.: Image compressed sensing using convolutional neural network. IEEE Trans. Image Process. 29, 375–388 (2019). https://doi.org/10.1109/TIP.2019.2928136

    Article  MathSciNet  MATH  Google Scholar 

  17. Zha, Z., Yuan, X., Zhou, J. T.: The power of triply complementary priors for image compressive sensing. In: 2020 IEEE International Conference on Image Processing, pp. 983–987. (2020)

  18. Iliadis, M., Spinoulas, L., Katsaggelos, A.K.: Deep fully-connected networks for video compressive sensing. Dig. Signal Process. 72, 9–18 (2018)

    Article  Google Scholar 

  19. Qiao, M., Meng, Z., Ma, J., Yuan, X.: Deep learning for video compressive sensing. APL Photonics 5(3), 030801 (2020)

    Article  Google Scholar 

  20. Wang, Z., Zhang, H., Cheng, Z., Chen, B., Yuan, X.: Metasci: Scalable and adaptive reconstruction for video compressive sensing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2083–2092 (2021)

  21. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision, pp. 694–711. Springer, Cham (2016)

  22. Chen, D., Yuan, L., Liao, J., Yu, N., Hua, G.: Stylebank: An explicit representation for neural image style transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1897–1906 (2017)

  23. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4681–4690 (2017)

  24. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition, Comput. Sci. (2014)

  25. Bora, A., Jalal, A., Price, E., Dimakis, A.G.: Compressed sensing using generative models. In: International Conference on Machine Learning, pp. 537–546. (2017)

  26. Kabkab, M., Samangouei, P., Chellappa, R.: Task-aware compressed sensing with generative adversarial networks. In: Proceedings of the AAAI Conference on Artificial Intelligence. (2018)

  27. Goodfellow, I.J.: Generative adversarial networks. Adv. Neural. Inf. Process. Syst. 3, 2672–2680 (2014)

    Google Scholar 

  28. Li, J., Fang, F., Mei, K., Zhang, G.: Multi-scale residual network for image super-resolution. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 517–532 (2018)

  29. Yeh, R. A., Chen, C., Yian Lim, T., Schwing, A. G., Hasegawa-Johnson, M., Do, M.N.: Semantic image inpainting with deep generative models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5485–5493 (2017)

  30. Shen, W., Liu, R.: Learning residual images for face attribute manipulation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4030–4038 (2017)

  31. Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. Computer ence. (2015)

  32. Timofte, R., Agustsson, E., Van Gool, L., Yang, M.H., Zhang, L.: Ntire 2017 challenge on single image super-resolution: Methods and results. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 114–125 (2017)

  33. Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: BMVC, pp. 26–135 (2012) http://dx.doi.org/https://doi.org/10.5244/C.26.135

  34. Zeyde, R., Elad, M., cProtter, M.: On single image scale-up using sparse-representations. In: International Conference on Curves and Surfaces, pp. 711–730 (2010)

  35. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE International Conference on Computer Vision, pp. 416–423 (2001)

  36. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv. 1412.6980 (2014)

  37. Shi, W., Jiang, F., Liu, S.: Scalable convolutional neural network for image compressed sensing. In: IEEE Comp Soc. pp. 12290–12299 (2019)

Download references

Funding

(Supported by the National Natural Science Foundation of China (Grant No. 61871261) and the Key Project of Science and Technology of Shanghai (Grant No. 19DZ1205802)).

Author information

Authors and Affiliations

  1. Department of Communication and Information Engineering, Shanghai University, Shanghai, China

    Jinpeng Tian, Wenjie Yuan & Yunxuan Tu

Authors
  1. Jinpeng Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wenjie Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yunxuan Tu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinpeng Tian.

Ethics declarations

Conflict of interest

The authors declared that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tian, J., Yuan, W. & Tu, Y. Image compressed sensing using multi-scale residual generative adversarial network. Vis Comput 38, 4193–4202 (2022). https://doi.org/10.1007/s00371-021-02288-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-021-02288-y

Keywords

Search

Navigation