Skip to main content
SpringerLink
Log in

Deep Memory-Augmented Proximal Unrolling Network for Compressive Sensing

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Mapping a truncated optimization method into a deep neural network, deep proximal unrolling network has attracted attention in compressive sensing due to its good interpretability and high performance. Each stage in such networks corresponds to one iteration in optimization. By understanding the network from the perspective of the human brain’s memory processing, we find there exist two categories of memory transmission: intra-stage and inter-stage. For intra-stage, existing methods increase the number of parameters to maximize the information flow. For inter-stage, there are also two methods. One is to transfer the information between adjacent stages, which can be regarded as short-term memory that is usually lost seriously. The other is a mechanism to ensure that the previous stages affect the current stage, which has not been explicitly studied. In this paper, a novel deep proximal unrolling network with persistent memory is proposed, dubbed deep Memory-Augmented Proximal Unrolling Network (MAPUN). We design a memory-augmented proximal mapping module that ensures maximum information flow for intra- and inter-stage. Specifically, we present a self-modulated block that can adaptively develop feature modulation for intra-stage and introduce two types of memory augmentation mechanisms for inter-stage, namely High-throughput Short-term Memory (HSM) and Cross-stage Long-term Memory (CLM). HSM is exploited to allow the network to transmit multi-channel short-term memory, which greatly reduces information loss between adjacent stages. CLM is utilized to develop the dependency of deep information across cascading stages, which greatly enhances network representation capability. Extensive experiments show that our MAPUN outperforms existing state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

References

  • Aggarwal, H. K., Mani, M. P., & Jacob, M. (2018). MoDL: Model-based deep learning architecture for inverse problems. IEEE Transactions on Medical Imaging, 38(2), 394–405.

    Article  Google Scholar 

  • Agustsson, E., & Timofte, R. (2017). Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

  • Buades, A., Coll, B., & Morel, J.-M. (2005). A non-local algorithm for image denoising. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

  • Chen, J., Sun, Y., Liu, Q., & Huang, R. (2020). Learning memory augmented cascading network for compressed sensing of images. In Proceedings of the European Conference on Computer Vision (ECCV).

  • Chen, Z., Hou, X., Gong, C., & Qian, X. (2016). Compressive sensing reconstruction for compressible signal based on projection replacement. Multimedia Tools and Applications, 75(5), 2565–2578.

    Article  Google Scholar 

  • Chen, Z., Hou, X., Qian, X., & Gong, C. (2017). Efficient and robust image coding and transmission based on scrambled block compressive sensing. IEEE Transactions on Multimedia, 20(7), 1610–1621.

    Google Scholar 

  • Chen, Y., & Pock, T. (2016). Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), 1256–1272.

    Article  Google Scholar 

  • Cichon, J., & Gan, W.-B. (2015). Branch-specific dendritic Ca 2+ spikes cause persistent synaptic plasticity. Nature, 520(7546), 180–185.

    Article  Google Scholar 

  • Dong, W., Wang, P., Yin, W., Shi, G., Wu, F., & Lu, X. (2018). Denoising prior driven deep neural network for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(10), 2305–2318.

    Article  Google Scholar 

  • Dong, W., Zhang, L., Shi, G., & Li, X. (2012). Nonlocally centralized sparse representation for image restoration. IEEE Transactions on Image Processing, 22(4), 1620–1630.

    Article  MathSciNet  MATH  Google Scholar 

  • Duarte, M. F., Davenport, M. A., Takhar, D., Laska, J. N., Sun, T., Kelly, K. F., & Baraniuk, R. G. (2008). Single-pixel imaging via compressive sampling. IEEE Signal Processing Magazine, 25(2), 83–91.

    Article  Google Scholar 

  • Elad, M., & Aharon, M. (2006). Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image processing, 15(12), 3736–3745.

    Article  MathSciNet  Google Scholar 

  • Fu, Y., Liang, Z., & You, S. (2021). Bidirectional 3D quasi-recurrent neural network for hyperspectral image super-resolution. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14, 2674–2688.

    Article  Google Scholar 

  • Fu, Y., Zhang, T., Wang, L., & Huang, H. (2022). Coded hyperspectral image reconstruction using deep external and internal learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7), 3404–3420.

    Google Scholar 

  • Gao, X., Zhang, J., Che, W., Fan, X., & Zhao, D. (2015). Block-based compressive sensing coding of natural images by local structural measurement matrix. In Proceedings of Data Compression Conference (DCC).

  • Gilton, D., Ongie, G., & Willett, R. (2019). Neumann networks for linear inverse problems in imaging. IEEE Transactions on Computational Imaging, 6, 328–343.

    Article  MathSciNet  Google Scholar 

  • Gu, J., & Dong, C. (2021). Interpreting super-resolution networks with local attribution maps. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

  • Hebb, D.O. (2005). The organization of behavior: A neuropsychological theory

  • Hyun, C. M., Kim, H. P., Lee, S. M., Lee, S., & Seo, J. K. (2018). Deep learning for undersampled MRI reconstruction. Physics in Medicine & Biology, 63(13), 135007.

    Article  Google Scholar 

  • Iliadis, M., Spinoulas, L., & Katsaggelos, A. K. (2018). Deep fully-connected networks for video compressive sensing. Digital Signal Processing, 72, 9–18.

    Article  Google Scholar 

  • Josselyn, S. A., & Tonegawa, S. (2020). Memory engrams: Recalling the past and imagining the future. Science, 367, 6473.

    Article  Google Scholar 

  • Kim, Y., Nadar, M.S., & Bilgin, A. (2010). Compressed sensing using a Gaussian scale mixtures model in wavelet domain. In Proceedings of the IEEE International Conference on Image Processing (ICIP).

  • Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR).

  • Kokkinos, F., & Lefkimmiatis, S. (2018). Deep image demosaicking using a cascade of convolutional residual denoising networks. In Proceedings of the European Conference on Computer Vision (ECCV).

  • Kruse, J., Rother, C., & Schmidt, U. (2017). Learning to push the limits of efficient FFT-based image deconvolution. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).

  • Kulkarni, K., Lohit, S., Turaga, P., Kerviche, R., & Ashok, A. (2016). Reconnet: Non-iterative reconstruction of images from compressively sensed measurements. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

  • Lai, Z., Wei, K., & Fu, Y. (2022). Deep plug-and-play prior for hyperspectral image restoration. Neurocomputing, 481, 281–293.

    Article  Google Scholar 

  • Lefkimmiatis, S. (2017). Non-local color image denoising with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

  • Li, X., Wu, J., Lin, Z., Liu, H., & Zha, H. (2018). Recurrent squeeze-and-excitation context aggregation net for single image deraining. In Proceedings of the European Conference on Computer Vision (ECCV).

  • Liu, J.-J., Hou, Q., Cheng, M.-M., Wang, C., & Feng, J. (2020). Improving convolutional networks with self-calibrated convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

  • Liu, P., Zhang, H., Zhang, K., Lin, L., & Zuo, W. (2018). Multi-level wavelet-CNN for image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

  • Liutkus, A., Martina, D., Popoff, S., Chardon, G., Katz, O., Lerosey, G., Gigan, S., Daudet, L., & Carron, I. (2014). Imaging with nature: Compressive imaging using a multiply scattering medium. Scientific Reports, 4, 5552.

    Article  Google Scholar 

  • Li, C., Yin, W., Jiang, H., & Zhang, Y. (2013). An efficient augmented Lagrangian method with applications to total variation minimization. Computational Optimization and Applications, 56(3), 507–530.

    Article  MathSciNet  MATH  Google Scholar 

  • Lustig, M., Donoho, D., & Pauly, J. M. (2007). Sparse MRI: The application of compressed sensing for rapid MR imaging. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine, 58(6), 1182–1195.

    Article  Google Scholar 

  • Mairal, J., Bach, F., Ponce, J., Sapiro, G., & Zisserman, A. (2009). Non-local sparse models for image restoration. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).

  • Mairal, J., Elad, M., & Sapiro, G. (2007). Sparse representation for color image restoration. IEEE Transactions on Image Processing, 17(1), 53–69.

    Article  MathSciNet  Google Scholar 

  • Metzler, C. A., Maleki, A., & Baraniuk, R. G. (2016). From denoising to compressed sensing. IEEE Transactions on Information Theory, 62(9), 5117–5144.

    Article  MathSciNet  MATH  Google Scholar 

  • Mousavi, A., Patel, A.B., & Baraniuk, R.G. (2015). A deep learning approach to structured signal recovery. In Proceedings of the Annual Allerton Conference on Communication, Control, and Computing (Allerton).

  • Ning, Q., Dong, W., Shi, G., Li, L., & Li, X. (2020). Accurate and lightweight image super-resolution with model-guided deep unfolding network. IEEE Journal of Selected Topics in Signal Processing, 15(2), 240–252.

    Article  Google Scholar 

  • Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., & Chintala, S. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Proceedings of Neural Information Processing Systems (NeurIPS).

  • Pian, Q., Yao, R., Sinsuebphon, N., & Intes, X. (2017). Compressive hyperspectral time-resolved wide-field fluorescence lifetime imaging. Nature Photonics, 11(7), 411–414.

    Article  Google Scholar 

  • Rousset, F., Ducros, N., Farina, A., Valentini, G., D’Andrea, C., & Peyrin, F. (2016). Adaptive basis scan by wavelet prediction for single-pixel imaging. IEEE Transactions on Computational Imaging, 3(1), 36–46.

    Article  MathSciNet  Google Scholar 

  • Sankaranarayanan, A. C., Studer, C., & Baraniuk, R. G. (2012). CS-MUVI: Video compressive sensing for spatial-multiplexing cameras. In Proceedings of the IEEE International Conference on Computational Photography (ICCP).

  • Schlemper, J., Caballero, J., Hajnal, J. V., Price, A. N., & Rueckert, D. (2017). A deep cascade of convolutional neural networks for dynamic MR image reconstruction. IEEE Transactions on Medical Imaging, 37(2), 491–503.

    Article  Google Scholar 

  • Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., & Woo, W.-c. (2015). Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS).

  • Shi, W., Jiang, F., Liu, S., & Zhao, D. (2019a). Scalable convolutional neural network for image compressed sensing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

  • Shi, W., Jiang, F., Liu, S., & Zhao, D. (2019b). Image compressed sensing using convolutional neural network. IEEE Transactions on Image Processing, 29, 375–388.

    Article  MathSciNet  MATH  Google Scholar 

  • Song, J., Chen, B., & Zhang, J. (2021). Memory-augmented deep unfolding network for compressive sensing. In Proceedings of the ACM International Conference on Multimedia (ACM MM).

  • Su, Y., & Lian, Q. (2020). iPiano-Net: Nonconvex optimization inspired multi-scale reconstruction network for compressed sensing. Signal Processing: Image Communication, 89, 115989.

    Google Scholar 

  • Sun, L., Fan, Z., Huang, Y., Ding, X., & Paisley, J.W. (2018). Compressed sensing MRI using a recursive dilated network. In Proceedings of the Conference on Association for the Advancement of Artificial Intelligence (AAAI).

  • Sun, Y., Chen, J., Liu, Q., Liu, B., & Guo, G. (2020). Dual-path attention network for compressed sensing image reconstruction. IEEE Transactions on Image Processing, 29, 9482–9495.

    Article  MATH  Google Scholar 

  • Sun, M.-J., Edgar, M. P., Gibson, G. M., Sun, B., Radwell, N., Lamb, R., & Padgett, M. J. (2016). Single-pixel three-dimensional imaging with time-based depth resolution. Nature Communications, 7(1), 1–6.

    Article  Google Scholar 

  • Szczykutowicz, T. P., & Chen, G. (2010). Dual energy CT using slow kVp switching acquisition and prior image constrained compressed sensing. Physics in Medicine & Biology, 55(21), 6411.

    Article  Google Scholar 

  • Tai, Y., Yang, J., Liu, X., & Xu, C. (2017). MemNet: A persistent memory network for image restoration. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).

  • Wang, X., Yu, K., Dong, C., & Loy, C.C. (2018). Recovering realistic texture in image super-resolution by deep spatial feature transform. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

  • Wang, H., Zhang, T., Yu, M., Sun, J., Ye, W., Wang, C., & Zhang, S. (2020). Stacking networks dynamically for image restoration based on the plug-and-play framework. In Proceedings of the European Conference on Computer Vision (ECCV).

  • Wu, Z., Zhang, J., & Mou, C. (2021a). Dense deep unfolding network with 3D-CNN prior for snapshot compressive sensing. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).

  • Wu, Z., Zhang, Z., Song, J., & Zhang, J. (2021b). Spatial-temporal synergic prior driven unfolding network for snapshot compressive imaging. In Proceedings of IEEE International Conference on Multimedia and Expo (ICME).

  • Yang, Y., Sun, J., Li, H., & Xu, Z. (2018). ADMM-CSNet: A deep learning approach for image compressive sensing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(3), 521–538.

    Article  Google Scholar 

  • You, D., Xie, J., & Zhang, J. (2021a). ISTA-Net++: Flexible deep unfolding network for compressive sensing. In Proceedings of IEEE International Conference on Multimedia and Expo (ICME).

  • You, D., Zhang, J., Xie, J., Chen, B., & Ma, S. (2021b). COAST: Controllable arbitrary-sampling network for compressive sensing. IEEE Transactions on Image Processing, 30, 6066–6080.

    Article  MathSciNet  Google Scholar 

  • Zhang, J., & Ghanem, B. (2018). ISTA-Net: Interpretable optimization-inspired deep network for image compressive sensing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

  • Zhang, K., Zuo, W., Gu, S., & Zhang, L. (2017). Learning deep CNN denoiser prior for image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

  • Zhang, Z., Jung, T.-P., Makeig, S., & Rao, B. D. (2012). Compressed sensing for energy-efficient wireless telemonitoring of noninvasive fetal ECG via block sparse bayesian learning. IEEE Transactions on Biomedical Engineering, 60(2), 300–309.

  • Zhang, Z., Liu, Y., Liu, J., Wen, F., & Zhu, C. (2020a). AMP-Net: Denoising-based deep unfolding for compressive image sensing. IEEE Transactions on Image Processing, 30, 1487–1500.

    Article  MathSciNet  Google Scholar 

  • Zhang, K., Li, Y., Zuo, W., Zhang, L., Van Gool, L., & Timofte, R. (2022). Plug-and-play image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10), 6360–6376.

    Article  Google Scholar 

  • Zhang, J., Zhao, D., & Gao, W. (2014a). Group-based sparse representation for image restoration. IEEE Transactions on Image Processing, 23(8), 3336–3351.

    Article  MathSciNet  MATH  Google Scholar 

  • Zhang, J., Zhao, C., & Gao, W. (2020b). Optimization-inspired compact deep compressive sensing. IEEE Journal of Selected Topics in Signal Processing, 14(4), 765–774.

    Article  Google Scholar 

  • Zhang, J., Zhao, C., Zhao, D., & Gao, W. (2014b). Image compressive sensing recovery using adaptively learned sparsifying basis via L0 minimization. Signal Processing, 103, 114–126.

    Article  Google Scholar 

  • Zhao, C., Ma, S., & Gao, W. (2014). Image compressive-sensing recovery using structured laplacian sparsity in DCT domain and multi-hypothesis prediction. In Proceedings of IEEE International Conference on Multimedia and Expo (ICME).

  • Zhao, C., Zhang, J., Ma, S., & Gao, W. (2016a). Nonconvex Lp nuclear norm based ADMM framework for compressed sensing. In Proceedings of Data Compression Conference (DCC).

  • Zhao, C., Ma, S., Zhang, J., Xiong, R., & Gao, W. (2016b). Video compressive sensing reconstruction via reweighted residual sparsity. IEEE Transactions on Circuits and Systems for Video Technology, 27(6), 1182–1195.

    Article  Google Scholar 

  • Zhao, C., Zhang, J., Wang, R., & Gao, W. (2018). CREAM: CNN-REgularized ADMM framework for compressive-sensed image reconstruction. IEEE Access, 6, 76838–76853.

    Article  Google Scholar 

  • Zha, Z., Wen, B., Yuan, X., Zhou, J., Zhu, C., & Kot, A. C. (2022). A hybrid structural sparsification error model for image restoration. IEEE Transactions on Neural Networks and Learning Systems, 33(9), 4451–4465.

    Article  MathSciNet  Google Scholar 

  • Zha, Z., Yuan, X., Wen, B., Zhou, J., & Zhu, C. (2020). Group sparsity residual constraint with non-local priors for image restoration. IEEE Transactions on Image Processing, 29, 8960–8975.

    Article  MathSciNet  MATH  Google Scholar 

  • Zheng, H., Fang, F., & Zhang, G. (2019). Cascaded dilated dense network with two-step data consistency for MRI reconstruction. In Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS).

  • Zhou, S., He, Y., Liu, Y., Li, C., & Zhang, J. (2021). Multi-channel deep networks for block-based image compressive sensing. IEEE Transactions on Multimedia, 23, 2627–2640.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Shenzhen Graduate School, Peking University, Shenzhen, China

    Jiechong Song, Bin Chen & Jian Zhang

  2. Peng Cheng Laboratory, Shenzhen, China

    Jian Zhang

Authors
  1. Jiechong Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jian Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jian Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Communicated by Ying Fu.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported in part by Shenzhen General Research Project (JCYJ20220531093215035).

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Song, J., Chen, B. & Zhang, J. Deep Memory-Augmented Proximal Unrolling Network for Compressive Sensing. Int J Comput Vis 131, 1477–1496 (2023). https://doi.org/10.1007/s11263-023-01765-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-023-01765-2

Keywords

Search

Navigation