Skip to main content
Springer Nature Link
Log in

Convolutional Neural Network Based on Multiple Attention Mechanisms for Hyperspectral and LiDAR Classification

  • Conference paper
  • First Online:
Spatial Data and Intelligence (SpatialDI 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14619))

Included in the following conference series:

  • 361 Accesses

Abstract

With the emergence of a large number of remote sensing data sources, how to effectively use the useful information in multi-source data for better earth observation has become an interesting but challenging problem. In this paper, the deep learning method is used to study the joint classification of hyperspectral imagery (HSI) and light detection and ranging (LiDAR) data. The network proposed in this paper is named convolutional neural network based on multiple attention mechanisms (MatNet). Specifically, a convolutional neural network (CNN) with an attention mechanism is used to extract the deep features of HSI and LiDAR respectively. Then the obtained features are introduced into the dual-branch cross-attention fusion module (DCFM) to fuse the information in HSI and LiDAR data effectively. Finally, the obtained features are introduced into the classification module to obtain the final classification results. Experimental results show that our proposed network can achieve better classification performance than existing methods.

This work is supported by the National Key R &D Program of China under Grant 2022YFF0503900.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Baltrušaitis, T., Ahuja, C., Morency, L.P.: Multimodal machine learning: a survey and taxonomy. IEEE Trans. Pattern Anal. Mach. Intell. 41(2), 423–443 (2019). https://doi.org/10.1109/TPAMI.2018.2798607

    Article  Google Scholar 

  2. Bartholomé, E., Belward, A.: GLC 2000: a new approach to global land cover mapping from earth observation data. Int. J. Remote Sens. 26, 1959–1977 (2005). https://doi.org/10.1080/01431160412331291297

    Article  Google Scholar 

  3. Chen, Y., et al.: Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3434–3443 (2019). https://doi.org/10.1109/ICCV.2019.00353

  4. Cheng, G., Yang, C., Yao, X., Guo, L., Han, J.: When deep learning meets metric learning: Remote sensing image scene classification via learning discriminative CNNs. IEEE Trans. Geosci. Remote Sens. 56(5), 2811–2821 (2018). https://doi.org/10.1109/TGRS.2017.2783902

    Article  Google Scholar 

  5. Debes, C., et al.: Hyperspectral and lidar data fusion: Outcome of the 2013 GRSS data fusion contest. IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens. 7 (2014). https://doi.org/10.1109/JSTARS.2014.2305441

  6. Hong, D., Gao, L., Hang, R., Zhang, B., Chanussot, J.: Deep encoder-decoder networks for classification of hyperspectral and lidar data. IEEE Geosci. Remote Sens. Lett. 19, 1–5 (2022). https://doi.org/10.1109/LGRS.2020.3017414

    Article  Google Scholar 

  7. Hong, D., et al.: More diverse means better: multimodal deep learning meets remote-sensing imagery classification. IEEE Trans. Geosci. Remote Sens. 59(5), 4340–4354 (2021). https://doi.org/10.1109/TGRS.2020.3016820

    Article  Google Scholar 

  8. Huang, R., Hong, D., Xu, Y., Yao, W., Stilla, U.: Multi-scale local context embedding for lidar point cloud classification. IEEE Geosci. Remote Sens. Lett. 17(4), 721–725 (2020). https://doi.org/10.1109/LGRS.2019.2927779

    Article  Google Scholar 

  9. Kang, J., Hong, D., Liu, J., Baier, G., Yokoya, N., Demir, B.: Learning convolutional sparse coding on complex domain for interferometric phase restoration. IEEE Trans. Neural Netw. Learn. Syst. 32(2), 826–840 (2021). https://doi.org/10.1109/TNNLS.2020.2979546

    Article  MathSciNet  Google Scholar 

  10. Li, H.C., Hu, W.S., Li, W., Li, J., Du, Q., Plaza, A.: A3 CLNN: spatial, spectral and multiscale attention ConvLSTM neural network for multisource remote sensing data classification. IEEE Trans. Neural Netw. Learn. Syst. 33(2), 747–761 (2022). https://doi.org/10.1109/TNNLS.2020.3028945

    Article  Google Scholar 

  11. Mohla, S., Pande, S., Banerjee, B., Chaudhuri, S.: FusAtNet: dual attention based spectrospatial multimodal fusion network for hyperspectral and lidar classification. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 416–425 (2020). https://doi.org/10.1109/CVPRW50498.2020.00054

  12. Qian, X., et al.: Generating and sifting pseudolabeled samples for improving the performance of remote sensing image scene classification. IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens. 13, 4925–4933 (2020). https://doi.org/10.1109/JSTARS.2020.3019582

    Article  Google Scholar 

  13. Rasti, B., et al.: Feature extraction for hyperspectral imagery: the evolution from shallow to deep: Overview and toolbox. IEEE Geosci. Remote Sens. Mag. 8(4), 60–88 (2020). https://doi.org/10.1109/MGRS.2020.2979764

    Article  Google Scholar 

  14. Turner, W., Spector, S., Gardiner, E., Fladeland, M., Sterling, E., Steininger, M.: Remote sensing for biodiversity science and conservation. Trends Ecol. Evol. 18, 306–314 (2003). https://doi.org/10.1016/S0169-5347(03)00070-3

    Article  Google Scholar 

  15. Ustin, S.: Manual of Remote Sensing/Remote Sensing for Natural Resource Management and Environmental Monitoring (2004)

    Google Scholar 

  16. Wang, M., Gao, F., Dong, J., Li, H.C., Du, Q.: Nearest neighbor-based contrastive learning for hyperspectral and lidar data classification. IEEE Trans. Geosci. Remote Sens. 61, 1–16 (2023). https://doi.org/10.1109/TGRS.2023.3236154

    Article  Google Scholar 

  17. Wang, W., Tran, D., Feiszli, M.: What makes training multi-modal classification networks hard? In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12692–12702 (2020). https://doi.org/10.1109/CVPR42600.2020.01271

  18. Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: CBAM: convolutional block attention module. ArXiv abs/1807.06521 (2018). https://api.semanticscholar.org/CorpusID:49867180

  19. Xu, X., Li, W., Ran, Q., Du, Q., Gao, L., Zhang, B.: Multisource remote sensing data classification based on convolutional neural network. IEEE Trans. Geosci. Remote Sens. 56(2), 937–949 (2018). https://doi.org/10.1109/TGRS.2017.2756851

    Article  Google Scholar 

  20. Zhang, B., Li, S., Jia, X., Gao, L., Peng, M.: Adaptive Markov random field approach for classification of hyperspectral imagery. IEEE Geosci. Remote Sens. Lett. 8(5), 973–977 (2011). https://doi.org/10.1109/LGRS.2011.2145353

    Article  Google Scholar 

  21. Zhang, C., Yang, Z., He, X., Deng, L.: Multimodal intelligence: representation learning, information fusion, and applications. IEEE J. Sel. Top. Signal Process. 14(3), 478–493 (2020). https://doi.org/10.1109/JSTSP.2020.2987728

    Article  Google Scholar 

  22. Zhang, M., Li, W., Du, Q., Gao, L., Zhang, B.: Feature extraction for classification of hyperspectral and lidar data using patch-to-patch CNN. IEEE Trans. Cybern. 50(1), 100–111 (2020). https://doi.org/10.1109/TCYB.2018.2864670

    Article  Google Scholar 

  23. Zhao, X., Tao, R., Li, W., Philips, W., Liao, W.: Fractional Gabor convolutional network for multisource remote sensing data classification. IEEE Trans. Geosci. Remote Sens. 60, 1–18 (2022). https://doi.org/10.1109/TGRS.2021.3065507

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Beijing University of Technology, Beijing, 100124, China

    Yingying Wang & Kun Wang

  2. Institute of Software, Chinese Academy of Sciences, Beijing, 100190, China

    Zhiming Ding

Authors
  1. Yingying Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhiming Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhiming Ding .

Editor information

Editors and Affiliations

  1. Renmin University of China, Beijing, China

    Xiaofeng Meng

  2. Nanjing Normal University, Nanjing, China

    Xueying Zhang

  3. Beijing University of Chemical Technology, Beijing, China

    Danhuai Guo

  4. Nanjing Normal University, Nanjing, China

    Di Hu

  5. Huazhong University of Science and Technology, Wuhan, China

    Bolong Zheng

  6. Hefei University of Technology, Hefei, China

    Chunju Zhang

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, Y., Wang, K., Ding, Z. (2024). Convolutional Neural Network Based on Multiple Attention Mechanisms for Hyperspectral and LiDAR Classification. In: Meng, X., Zhang, X., Guo, D., Hu, D., Zheng, B., Zhang, C. (eds) Spatial Data and Intelligence. SpatialDI 2024. Lecture Notes in Computer Science, vol 14619. Springer, Singapore. https://doi.org/10.1007/978-981-97-2966-1_20

Download citation

  • DOI: https://doi.org/10.1007/978-981-97-2966-1_20

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-97-2965-4

  • Online ISBN: 978-981-97-2966-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics