Skip to main content
SpringerLink
Log in

Ensemble Learning Priors Driven Deep Unfolding for Scalable Video Snapshot Compressive Imaging

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13683))

Included in the following conference series:

Abstract

Snapshot compressive imaging (SCI) can record a 3D datacube by a 2D measurement and algorithmically reconstruct the desired 3D information from that 2D measurement. The reconstruction algorithm thus plays a vital role in SCI. Recently, deep learning (DL) has demonstrated outstanding performance in reconstruction, leading to better results than conventional optimization-based methods. Therefore, it is desirable to improve DL reconstruction performance for SCI. Existing DL algorithms are limited by two bottlenecks: 1) a high-accuracy network is usually large and requires a long running time; 2) DL algorithms are limited by scalability, i.e., a well-trained network cannot generally be applied to new systems. To this end, this paper proposes to use ensemble learning priors in DL to achieve high reconstruction speed and accuracy in a single network. Furthermore, we develop the scalable learning approach during training to empower DL to handle data of different sizes without additional training. Extensive results on both simulation and real datasets demonstrate the superiority of our proposed algorithm. The code and model can be accessed at https://github.com/integritynoble/ELP-Unfolding/tree/master.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Afonso, M.V., Bioucas-Dias, J.M., Figueiredo, M.A.: An augmented Lagrangian approach to the constrained optimization formulation of imaging inverse problems. IEEE Trans. Image Process. 20(3), 681–695 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  2. Akkus, Z., Galimzianova, A., Hoogi, A., Rubin, D.L., Erickson, B.J.: Deep learning for brain MRI segmentation: state of the art and future directions. J. Digit. Imaging 30(4), 449–459 (2017)

    Article  Google Scholar 

  3. Antipa, N., Oare, P., Bostan, E., Ng, R., Waller, L.: Video from stills: lensless imaging with rolling shutter. In: 2019 IEEE International Conference on Computational Photography (ICCP), pp. 1–8. IEEE (2019)

    Google Scholar 

  4. Bioucas-Dias, J.M., Figueiredo, M.A.: A new twist: two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. Image Process. 16(12), 2992–3004 (2007)

    Article  MathSciNet  Google Scholar 

  5. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)

    Article  MATH  Google Scholar 

  6. Cai, Y., et al.: Mask-guided spectral-wise transformer for efficient hyperspectral image reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022)

    Google Scholar 

  7. Chen, Z., Zheng, S., Tong, Z., Yuan, X.: Physics-driven deep learning enables temporal compressive coherent diffraction imaging. Optica 9(6), 677–680 (2022)

    Article  Google Scholar 

  8. Cheng, Z., Chen, B., Liu, G., Zhang, H., Lu, R., Wang, Z., Yuan, X.: Memory-efficient network for large-scale video compressive sensing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16246–16255 (2021)

    Google Scholar 

  9. Cheng, Z., Chen, B., Lu, R., Wang, Z., Zhang, H., Meng, Z., Yuan, X.: Recurrent neural networks for snapshot compressive imaging. In: IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)

    Google Scholar 

  10. Cheng, Z., Lu, R., Wang, Z., Zhang, H., Chen, B., Meng, Z., Yuan, X.: BIRNAT: bidirectional recurrent neural networks with adversarial training for video snapshot compressive imaging. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12369, pp. 258–275. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58586-0_16

    Chapter  Google Scholar 

  11. Dong, J., Fu, J., He, Z.: A deep learning reconstruction framework for x-ray computed tomography with incomplete data. PLoS ONE 14(11), e0224426 (2019)

    Article  Google Scholar 

  12. Dong, W., Shi, G., Li, X., Ma, Y., Huang, F.: Compressive sensing via nonlocal low-rank regularization. IEEE Trans. Image Process. 23(8), 3618–3632 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  13. Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  14. Duarte, M.F., et al.: Single-pixel imaging via compressive sampling. IEEE Signal Process. Mag. 25(2), 83–91 (2008)

    Article  MathSciNet  Google Scholar 

  15. Emmanuel, C., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  16. Gibson, G.M., et al.: Real-time imaging of methane gas leaks using a single-pixel camera. Opt. Express 25(4), 2998–3005 (2017)

    Article  Google Scholar 

  17. Gregor, K., LeCun, Y.: Learning fast approximations of sparse coding. In: Proceedings of the 27th International Conference on Machine Learning, pp. 399–406 (2010)

    Google Scholar 

  18. He, W., Yokoya, N., Yuan, X.: Fast hyperspectral image recovery via non-iterative fusion of dual-camera compressive hyperspectral imaging. IEEE Trans. Image Process. 30, 1–12 (2021)

    Article  MathSciNet  Google Scholar 

  19. Higham, C.F., Murray-Smith, R., Padgett, M.J., Edgar, M.P.: Deep learning for real-time single-pixel video. Sci. Rep. 8(1), 1–9 (2018)

    Article  Google Scholar 

  20. Hitomi, Y., Gu, J., Gupta, M., Mitsunaga, T., Nayar, S.K.: Video from a single coded exposure photograph using a learned over-complete dictionary. In: 2011 International Conference on Computer Vision, pp. 287–294. IEEE (2011)

    Google Scholar 

  21. Hu, X., Cai, Y., Lin, J., Wang, H., Yuan, X., Zhang, Y., Timofte, R., Van Gool, L.: Hdnet: High-resolution dual-domain learning for spectral compressive imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022)

    Google Scholar 

  22. Jalali, S., Yuan, X.: Snapshot compressed sensing: performance bounds and algorithms. IEEE Trans. Inf. Theory 65(12), 8005–8024 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  23. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015, Conference Track Proceedings (2015)

    Google Scholar 

  24. Kulkarni, K., Lohit, S., Turaga, P., Kerviche, R., Ashok, A.: Reconnet: Non-iterative reconstruction of images from compressively sensed measurements. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 449–458 (2016)

    Google Scholar 

  25. Li, C.: An Efficient Algorithm For Total Variation Regularization with Applications to the Single Pixel Camera and Compressive Sensing. Rice University (2010)

    Google Scholar 

  26. Li, C., Yin, W., Jiang, H., Zhang, Y.: An efficient augmented Lagrangian method with applications to total variation minimization. Comput. Optim. Appl. 56(3), 507–530 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  27. Lin, J., et al.: Coarse-to-fine sparse transformer for hyperspectral image reconstruction. arXiv preprint arXiv:2203.04845 (2022)

  28. Liu, J., et al.: Applications of deep learning to MRI images: a survey. Big Data Mining Anal. 1(1), 1–18 (2018)

    Article  Google Scholar 

  29. Liu, Y., Yuan, X., Suo, J., Brady, D.J., Dai, Q.: Rank minimization for snapshot compressive imaging. IEEE Trans. Pattern Anal. Mach. Intell. 41(12), 2990–3006 (2018)

    Article  Google Scholar 

  30. Llull, P., Liao, X., Yuan, X., Yang, J., Kittle, D., Carin, L., Sapiro, G., Brady, D.J.: Coded aperture compressive temporal imaging. Opt. Express 21(9), 10526–10545 (2013)

    Article  Google Scholar 

  31. Meng, Z., Jalali, S., Yuan, X.: Gap-net for snapshot compressive imaging. arXiv preprint arXiv:2012.08364 (2020)

  32. Meng, Z., Ma, J., Yuan, X.: End-to-end low cost compressive spectral imaging with spatial-spectral self-attention. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12368, pp. 187–204. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58592-1_12

    Chapter  Google Scholar 

  33. Mercat, A., Viitanen, M., Vanne, J.: Uvg dataset: 50/120fps 4k sequences for video codec analysis and development. In: Proceedings of the 11th ACM Multimedia Systems Conference, pp. 297–302 (2020)

    Google Scholar 

  34. Mousavi, A., Patel, A.B., Baraniuk, R.G.: A deep learning approach to structured signal recovery. In: 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 1336–1343. IEEE (2015)

    Google Scholar 

  35. Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. Adv. Neural. Inf. Process. Syst. 32, 8026–8037 (2019)

    Google Scholar 

  36. Pintelas, P., Livieris, I.E.: Special issue on ensemble learning and applications. Algorithms 13(6), 140 (2020)

    Article  MathSciNet  Google Scholar 

  37. Pont-Tuset, J., Perazzi, F., Caelles, S., Arbeláez, P., Sorkine-Hornung, A., Van Gool, L.: The 2017 Davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675 (2017)

  38. Qiao, M., Meng, Z., Ma, J., Yuan, X.: Deep learning for video compressive sensing. APL Photon. 5(3), 030801 (2020)

    Article  Google Scholar 

  39. Qiao, M., Sun, Y., Ma, J., Meng, Z., Liu, X., Yuan, X.: Snapshot coherence tomographic imaging. IEEE Trans. Comput. Imaging 7, 624–637 (2021)

    Article  MathSciNet  Google Scholar 

  40. Radwell, N., et al.: Deep learning optimized single-pixel lidar. Appl. Phys. Lett. 115(23), 231101 (2019)

    Article  Google Scholar 

  41. Reddy, D., Veeraraghavan, A., Chellappa, R.: P2c2: programmable pixel compressive camera for high speed imaging. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 329–336 (2011)

    Google Scholar 

  42. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  43. Shi, W., Jiang, F., Zhang, S., Zhao, D.: Deep networks for compressed image sensing. In: 2017 IEEE International Conference on Multimedia and Expo (ICME), pp. 877–882. IEEE (2017)

    Google Scholar 

  44. Tassano, M., Delon, J., Veit, T.: FastDVDNet: towards real-time deep video denoising without flow estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1354–1363 (2020)

    Google Scholar 

  45. Wang, Z., Zhang, H., Cheng, Z., Chen, B., Yuan, X.: MetaSci: scalable and adaptive reconstruction for video compressive sensing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2083–2092 (2021)

    Google Scholar 

  46. Wu, Z., Zhang, J., Mou, C.: Dense deep unfolding network with 3d-CNN prior for snapshot compressive imaging. In: IEEE International Conference on Computer Vision (ICCV) (2021)

    Google Scholar 

  47. Xue, Y., et al.: Block modulating video compression: an ultra low complexity image compression encoder for resource limited platforms. CoRR (2022)

    Google Scholar 

  48. Yang, C., et al.: Improving the image reconstruction quality of compressed ultrafast photography via an augmented Lagrangian algorithm. J. Opt. 21(3), 035703 (2019)

    Article  Google Scholar 

  49. Yang, C., et al.: High-fidelity image reconstruction for compressed ultrafast photography via an augmented-Lagrangian and deep-learning hybrid algorithm. Photon. Res. 9(2), B30–B37 (2021)

    Article  Google Scholar 

  50. Yang, Y., Sun, J., Li, H., Xu, Z.: Deep ADMM-net for compressive sensing MRI. In: Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 10–18 (2016)

    Google Scholar 

  51. Yang, Y., Sun, J., Li, H., Xu, Z.: ADMM-CSNET: a deep learning approach for image compressive sensing. IEEE Trans. Pattern Anal. Mach. Intell. 42(3), 521–538 (2018)

    Article  Google Scholar 

  52. Yoo, J., Sabir, S., Heo, D., Kim, K.H., Wahab, A., Choi, Y., Lee, S.I., Chae, E.Y., Kim, H.H., Bae, Y.M., et al.: Deep learning diffuse optical tomography. IEEE Trans. Med. Imaging 39(4), 877–887 (2019)

    Article  Google Scholar 

  53. Yuan, X.: Compressive dynamic range imaging via Bayesian shrinkage dictionary learning. Opt. Eng. 55(12), 123110 (2016)

    Article  Google Scholar 

  54. Yuan, X., Liao, X., Llull, P., Brady, D., Carin, L.: Efficient patch-based approach for compressive depth imaging. Appl. Opt. 55(27), 7556–7564 (2016)

    Article  Google Scholar 

  55. Yuan, X.: Generalized alternating projection based total variation minimization for compressive sensing. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 2539–2543 (2016)

    Google Scholar 

  56. Yuan, X., Brady, D.J., Katsaggelos, A.K.: Snapshot compressive imaging: theory, algorithms, and applications. IEEE Signal Process. Mag. 38(2), 65–88 (2021)

    Article  Google Scholar 

  57. Yuan, X., Jiang, H., Huang, G., Wilford, P.A.: Slope: shrinkage of local overlapping patches estimator for lensless compressive imaging. IEEE Sens. J. 16(22), 8091–8102 (2016)

    Article  Google Scholar 

  58. Yuan, X., Liu, Y., Suo, J., Dai, Q.: Plug-and-play algorithms for large-scale snapshot compressive imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1447–1457 (2020)

    Google Scholar 

  59. Yuan, X., Liu, Y., Suo, J., Durand, F., Dai, Q.: Plug-and-play algorithms for video snapshot compressive imaging. In: IEEE Transactions on Pattern Analysis and Machine Intelligence (2021)

    Google Scholar 

  60. Yuan, X., et al.: Low-cost compressive sensing for color video and depth. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3318–3325 (2014)

    Google Scholar 

  61. Yuan, X., Pu, Y.: Parallel lensless compressive imaging via deep convolutional neural networks. Opt. Express 26(2), 1962–1977 (2018)

    Article  Google Scholar 

  62. Zhang, B., Yuan, X., Deng, C., Zhang, Z., Suo, J., Dai, Q.: End-to-end snapshot compressed super-resolution imaging with deep optics. Optica 9(4), 451–454 (2022)

    Article  Google Scholar 

  63. Zhang, J., Ghanem, B.: ISTA-Net: interpretable optimization-inspired deep network for image compressive sensing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1828–1837 (2018)

    Google Scholar 

  64. Zhang, K., Zuo, W., Zhang, L.: FFDNet: toward a fast and flexible solution for CNN-based image denoising. IEEE Trans. Image Process. 27(9), 4608–4622 (2018)

    Article  MathSciNet  Google Scholar 

  65. Zheng, H., et al.: A new ensemble learning framework for 3d biomedical image segmentation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 5909–5916 (2019)

    Google Scholar 

  66. Zheng, S., et al.: Deep plug-and-play priors for spectral snapshot compressive imaging. Photon. Res. 9(2), B18–B29 (2021)

    Article  Google Scholar 

  67. Zheng, S., Wang, C., Yuan, X., Xin, H.L.: Super-compression of large electron microscopy time series by deep compressive sensing learning. Patterns 2(7), 100292 (2021)

    Article  Google Scholar 

  68. Zhou, K., Yang, Y., Qiao, Y., Xiang, T.: Domain adaptive ensemble learning. IEEE Trans. Image Process. 30, 8008–8018 (2021)

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank the Research Center for Industries of the Future (RCIF) at Westlake University, Westlake Foundation (2021B1501-2) and the funding from Lochn Optics.

Author information

Authors and Affiliations

  1. School of Engineering, Westlake University, Hangzhou, 310030, Zhejiang, China

    Chengshuai Yang, Shiyu Zhang & Xin Yuan

Authors
  1. Chengshuai Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shiyu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xin Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xin Yuan .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 6205 KB)

Supplementary material 2 (mp4 7193 KB)

Supplementary material 3 (mp4 1498 KB)

Supplementary material 4 (mp4 1674 KB)

Supplementary material 5 (mp4 704 KB)

Supplementary material 6 (mp4 483 KB)

Supplementary material 7 (mp4 508 KB)

Supplementary material 8 (mp4 575 KB)

Supplementary material 9 (pdf 1573 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yang, C., Zhang, S., Yuan, X. (2022). Ensemble Learning Priors Driven Deep Unfolding for Scalable Video Snapshot Compressive Imaging. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13683. Springer, Cham. https://doi.org/10.1007/978-3-031-20050-2_35

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20050-2_35

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20049-6

  • Online ISBN: 978-3-031-20050-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation