Skip to main content
SpringerLink
Log in

A Deep Wavelet Network for High-Resolution Microscopy Hyperspectral Image Reconstruction

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 Workshops (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13807))

Included in the following conference series:

  • 1632 Accesses

Abstract

Microscopy hyperspectral imaging (MHSI) integrates conventional imaging with spectroscopy to capture images through numbers of narrow spectral bands, and has attracted much attention in histopathology image analysis. However, due to the current hardware limitation, the observed microscopy hyperspectral images (MHSIs) has much lower spatial resolution than the coupled RGB images. High-resolution MHSI (HR-MHSI) reconstruction by fusing low-resolution MHSIs (LR-MHSIs) with high-resolution (HR) RGB images can compensate for the information loss. Unfortunately, radiometrical inconsistency between the two modalities often causes spectral and spatial distortions which degrade quality of the reconstructed images. To address these issues, we propose a Deep Wavelet Network (WaveNet) based on spectral grouping and feature adaptation for HR-MHSI reconstruction. For preservation of spectral information, the band-group-wise adaptation framework exploits intrinsic spectral correlation and decompose MHSI reconstruction in separate spectral groups. In spatial domain, we design the mutual adaptation block to jointly fuse the wavelet and convolution features by wavelet-attention for high-frequency information extraction. Specifically, to effectively inject spatial details from HR-RGB to LR-MHSI, radiometry-aware alignment is conducted between the two modalities. We compare WaveNet with several state-of-the-art methods by their performance on a public pathological MHSI dataset. Experimental results demonstrate its efficacy and reliability under different settings.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Arun, P.V., Buddhiraju, K.M., Porwal, A., Chanussot, J.: CNN-based super-resolution of hyperspectral images. IEEE Trans. Geosci. Remote Sens. 58(9), 6106–6121 (2020)

    Article  Google Scholar 

  2. Chen, Z., Guo, X., Yang, C., Ibragimov, B., Yuan, Y.: Joint spatial-wavelet dual-stream network for super-resolution. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12265, pp. 184–193. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59722-1_18

    Chapter  Google Scholar 

  3. Dian, R., Li, S., Fang, L.: Learning a low tensor-train rank representation for hyperspectral image super-resolution. IEEE Trans. Neural Netw. Learn. Syst. 30(9), 2672–2683 (2019)

    Article  MathSciNet  Google Scholar 

  4. Dong, W., et al.: Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Trans. Image Process. 25(5), 2337–2352 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  5. Dong, W., Zhou, C., Wu, F., Wu, J., Shi, G., Li, X.: Model-guided deep hyperspectral image super-resolution. IEEE Trans. Image Process. 30, 5754–5768 (2021)

    Article  Google Scholar 

  6. Dremin, V.: Skin complications of diabetes mellitus revealed by polarized hyperspectral imaging and machine learning. IEEE Trans. Med. Imaging 40(4), 1207–1216 (2021)

    Article  Google Scholar 

  7. Fang, L., Zhuo, H., Li, S.: Super-resolution of hyperspectral image via superpixel-based sparse representation. Neurocomputing 273, 171–177 (2018)

    Article  Google Scholar 

  8. Guo, T., Seyed Mousavi, H., Huu Vu, T., Monga, V.: Deep wavelet prediction for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 104–113 (2017)

    Google Scholar 

  9. Han, X.H., Shi, B., Zheng, Y.: Self-similarity constrained sparse representation for hyperspectral image super-resolution. IEEE Trans. Image Process. 27(11), 5625–5637 (2018)

    Article  MathSciNet  Google Scholar 

  10. Han, X.H., Zheng, Y., Chen, Y.W.: Multi-level and multi-scale spatial and spectral fusion CNN for hyperspectral image super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (2019)

    Google Scholar 

  11. Kanatsoulis, C.I., Fu, X., Sidiropoulos, N.D., Ma, W.K.: Hyperspectral super-resolution: a coupled tensor factorization approach. IEEE Trans. Sig. Process. 66(24), 6503–6517 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  12. Lanaras, C., Baltsavias, E.: Hyperspectral super-resolution by coupled spectral unmixing, pp. 3586–3594 (2015). https://doi.org/10.1109/ICCV.2015.409

  13. Liu, P., Zhang, H., Zhang, K., Lin, L., Zuo, W.: Multi-level wavelet-CNN for image restoration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 773–782 (2018)

    Google Scholar 

  14. Liu, X., Liu, Q., Wang, Y.: Remote sensing image fusion based on two-stream fusion network. Inf. Fusion 55, 1–15 (2020)

    Article  Google Scholar 

  15. Ma, W., Pan, Z., Guo, J., Lei, B.: Achieving super-resolution remote sensing images via the wavelet transform combined with the recursive res-net. IEEE Trans. Geosci. Remote Sens. 57(6), 3512–3527 (2019)

    Article  Google Scholar 

  16. Pan, X.,et al.: On the integration of self-attention and convolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 815–825 (2022)

    Google Scholar 

  17. Sun, L., et al.: Diagnosis of cholangiocarcinoma from microscopic hyperspectral pathological dataset by deep convolution neural networks. Methods 202, 22–30 (2022)

    Article  Google Scholar 

  18. Wang, H., Wu, X., Huang, Z., Xing, E.P.: High-frequency component helps explain the generalization of convolutional neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8684–8694 (2020)

    Google Scholar 

  19. Wang, Q., Li, Q., Li, X.: Hyperspectral image superresolution using spectrum and feature context. IEEE Trans. Industr. Electron. 68(11), 11276–11285 (2020)

    Article  Google Scholar 

  20. Wang, Q., et al.: Identification of melanoma from hyperspectral pathology image using 3D convolutional networks. IEEE Trans. Med. Imaging 40(1), 218–227 (2020)

    Article  Google Scholar 

  21. Wang, W., et al.: Enhanced deep blind hyperspectral image fusion. IEEE Trans. Neural Netw. Learn. Syst.(2021)

    Google Scholar 

  22. Wang, W., Zeng, W., Huang, Y., Ding, X., Paisley, J.: Deep blind hyperspectral image fusion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4150–4159 (2019)

    Google Scholar 

  23. Xie, Q., Zhou, M., Zhao, Q., Xu, Z., Meng, D.: MHF-net: an interpretable deep network for multispectral and hyperspectral image fusion. IEEE Trans. Pattern Anal. Mach. Intell. (2020)

    Google Scholar 

  24. Xin, J., Li, J., Jiang, X., Wang, N., Huang, H., Gao, X.: Wavelet-based dual recursive network for image super-resolution. IEEE Trans. Neural Netw. Learn. Syst. (2020)

    Google Scholar 

  25. Zhang, H., Ma, J.: GTP-Pnet: a residual learning network based on gradient transformation prior for pansharpening. ISPRS J. Photogramm. Remote. Sens. 172, 223–239 (2021)

    Article  Google Scholar 

  26. Zhang, X., Huang, W., Wang, Q., Li, X.: SSR-net: spatial-spectral reconstruction network for hyperspectral and multispectral image fusion. IEEE Trans. Geosci. Remote Sens. 59(7), 5953–5965 (2020)

    Article  Google Scholar 

  27. Zhang, Y., Wang, Y., Zhang, B., Li, Q.: A hyperspectral dataset of precancerous lesions in gastric cancer and benchmarkers for pathological diagnosis. J. Biophotonics (2022, in press). https://doi.org/10.1002/jbio.202200163

  28. Zheng, Y., Li, J., Li, Y., Guo, J., Wu, X., Chanussot, J.: Hyperspectral pansharpening using deep prior and dual attention residual network. IEEE Trans. Geosci. Remote Sens. 58(11), 8059–8076 (2020)

    Article  Google Scholar 

  29. Zhong, Z., Shen, T., Yang, Y., Lin, Z., Zhang, C.: Joint sub-bands learning with clique structures for wavelet domain super-resolution. In: Advances in Neural Information Processing Systems, vol. 31 (2018)

    Google Scholar 

  30. Zhu, Z., Hou, J., Chen, J., Zeng, H., Zhou, J.: Hyperspectral image super-resolution via deep progressive zero-centric residual learning. IEEE Trans. Image Process. 30, 1423–1438 (2020)

    Article  Google Scholar 

Download references

Acknowledgment

This work was supported in part by the “Chenguang Program” supported by Shanghai Education Development Foundation and Shanghai Municipal Education Commission under Grant 18CG38, the Shanghai Key Laboratory of Multidimensional Information Processing of the East China Normal University under Grant MIP20224 and the Fundamental Research Funds for the Central Universities under Grant 22D111211.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Donghua University, Shanghai, China

    Qian Wang & Zhao Chen

  2. Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University, Shanghai, China

    Qian Wang & Zhao Chen

Authors
  1. Qian Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhao Chen .

Editor information

Editors and Affiliations

  1. IBM Research - MIT-IBM Watson AI Lab, Massachusetts, USA

    Leonid Karlinsky

  2. Technion – Israel Institute of Technology, Haifa, Israel

    Tomer Michaeli

  3. Kyoto University, Kyoto, Japan

    Ko Nishino

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, Q., Chen, Z. (2023). A Deep Wavelet Network for High-Resolution Microscopy Hyperspectral Image Reconstruction. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13807. Springer, Cham. https://doi.org/10.1007/978-3-031-25082-8_44

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25082-8_44

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25081-1

  • Online ISBN: 978-3-031-25082-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation