Combined Classification of Hyperspectral and LiDAR Data based on Dual-Channel Cross-Transformer
Abstract
In the face of complex scenes, single-modal dominant classification tasks encounter limitations in performance due to insufficient information. On the other hand, joint classification of multimodal remote sensing data faces challenges such as data sample differences and lack of correlation in physical features between modalities, which can impact classification performance. To fully integrate the heterogeneous information of multimodal data and improve classification performance, we propose a dual-channel cross-transformer feature fusion extraction network. The framework leverages self-attention mechanisms to aggregate features within each modality, and the feature fusion module based on cross-modal attention fully considers the complementary information between modalities. Classification tasks are performed using the fused spatial-spectral features obtained from the joint representation of modalities. Extensive experiments conducted on the Houston and MUUFL datasets demonstrate the effectiveness of the proposed model compared to existing methods.
References
[1]
Z. Lin, Y. Chen, P. Ghamisi, and J. A. Benediktsson, “Generative adversarial networks for hyperspectral image classification,” IEEE Trans. Geosci.Remote Sens., vol. 56, no. 9, pp. 5046–5063, Sep. 2018.
[2]
P. Ghamisi, “Advances in hyperspectral image and signal processing: A comprehensive overview of the state of the art,” IEEE Geosci.Remote Sens. Mag., vol. 5, no. 4, pp. 37–78, Dec. 2017.
[3]
D. Hong, “Interpretable hyperspectral artificial intelligence: When nonconvex modeling meets hyperspectral remote sensing,” IEEE Geosci.Remote Sens. Mag., vol. 9, no. 2, pp. 52–87, Jun. 2021.
[4]
Z. Wang and M. Menenti, “Challenges and opportunities in LiDAR remote sensing,” Frontiers Remote Sens., vol. 2, Mar. 2021, Art. no. 641723.
[5]
W. Li, Y. Gao, M. Zhang, R. Tao, and Q. Du, “Asymmetric feature fusion network for hyperspectral and SAR image classification,” IEEE Trans. Neural Netw. Learn. Syst., early access, Feb. 18, 2022.
[6]
U. Heiden, W. Heldens, S. Roessner, K. Segl, T. Esch, and A. Mueller, “Urban structure type characterization using hyperspectral remote sensing and height information,” Landscape Urban Planning, vol. 105, no. 4, pp. 361–375, Apr. 2012.
[7]
Bakhshi A, Ghassemian M H, AhmadiFard A. The Effect of The Spectral Band Selection Method on Hyperspectral Images segmentation Accuracy[J]. International Journal of Computer Theory and Engineering, 2012, 4(3): 318.
[8]
Y. Chen, C. Li, P. Ghamisi, X. Jia, and Y. Gu, “Deep fusion of remotesensing data for accurate classification,” IEEE Geosci. Remote Sens. Lett.vol. 14, no. 8, pp. 1253–1257, Aug. 2017.
[9]
X. Xu, W. Li, Q. Ran, Q. Du, L. Gao, and B. Zhang, “Multisource remote sensing data classification based on convolutional neural network,” IEEE Trans. Geosci. Remote Sens., vol. 56, no. 2, pp.937–949, Feb. 2018.
[10]
He X, Chen Y, Lin Z. Spatial-spectral Transformer for Hyperspectral Image Classification[J]. Remote Sensing, 2021, 13(3): 498.
[11]
S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah,"Transformers in vision: A survey, " ACM Computing Surveys (CSUR),2021.
[12]
Y. Qing, W. Liu, L. Feng, and W. Gao, “Improved transformer net for hyperspectral image classification,” Remote Sens., vol. 13, no. 11, p. 2216, Jun. 2021.
[13]
D. Hong, “SpectralFormer: Rethinking hyperspectral image classification with transformers,” IEEE Trans. Geosci. Remote Sens., vol. 60, Nov. 2022, Art. no. 5518615.
[14]
S. K. Roy, A. Deria, D. Hong, B. Rasti, A. Plaza, and J. Chanussot, “Multimodal fusion transformer for remote sensing image classification,” 2022, arXiv:2203.16952.
[15]
G. Zhao, Q. Ye, L. Sun, Z. Wu, C. Pan and B. Jeon, "Joint Classification of Hyperspectral and LiDAR Data Using a Hierarchical CNN and Transformer," in IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1-16, 2023, Art no. 5500716.
[16]
Y. Zhang, Y. Peng, B. Tu and Y. Liu, "Local Information Interaction Transformer for Hyperspectral and LiDAR Data Classification," in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 16, pp. 1130-1143, 2023.
[17]
C. Debes, A. Merentitis, R. Heremans, “Hyperspectral and LiDAR Data Fusion: Outcome of the 2013 grss Data Fusion Contest,” IEEE J. Sel. Topics Appl.Earth Observ. Remote Sens., vol. 7, no. 6, pp. 2405–2418, 2014.
[18]
P. Gader, A. Zare, R. Close, J. Aitken, and G. Tuell, “MUUFL Gulfport hyperspectral and LiDAR airborne data set,” Tech. Rep. Rep. REP-2013-570, University of Florida, Gainesville, FL, Oct. 2013.
[19]
X. Du and A. Zare, “Technical report: scene label ground truth map for MUUFL Gulfport data set,”Tech. Rep. Tech. Rep. 20170417, University of Florida,Gainesville, FL., April 2017.
[20]
D. Hong, L. Gao, R. Hang, B. Zhang, and J. Chanussot, “Deep encoder–decoder networks for classification of hyperspectral and LiDAR data,” IEEE Geosci. Remote Sens. Lett., vol. 19, Aug. 2022, Art. no. 5500205.
[21]
T. Zhang, S. Xiao, W. Dong, J. Qu, and Y. Yang, “Amutual guidance attention-based multi-level fusion network for hyperspectral and LiDAR classification,” IEEE Geosci. Remote Sens. Lett., pp. 1–5, 2021.
Index Terms
- Combined Classification of Hyperspectral and LiDAR Data based on Dual-Channel Cross-Transformer
Recommendations
Coupled adversarial learning for fusion classification of hyperspectral and LiDAR data
AbstractHyperspectral image (HSI) provides rich spectral–spatial information and the light detection and ranging (LiDAR) data reflect the elevation information, which can be jointly exploited for better land-cover classification. However, due ...
Highlights- Propose a novel coupled adversarial learning method for multi-modal classification.
Reciprocal transformer for hyperspectral and multispectral image fusion
AbstractHyperspectral and multispectral (HS–MS) image fusion aims to reconstruct high-resolution hyperspectral images from low-resolution hyperspectral images and high-resolution multispectral images. While convolutional neural networks have been widely ...
Highlights- We advance a novel hyperspectral and multispectral image fusion method through dual cross Transformer model.
- The proposed method can reciprocally inject the spectral and spatial information to the target hyperspectral image.
- It ...
Cross-scale cascade transformer for multimodal human action recognition
Highlights- A cross-modal and cross-scale fusion module is proposed to perform multimodal feature interaction.
- The proposed fusion network can handle different multimodal input combinations and obtain significant performance improvement.
- ...
AbstractHuman action recognition can benefit from multimodal information to address the classification problem under complex situations. However, existing works either use score fusion or perform simple feature integration methods to combine multiple ...
Comments
Information & Contributors
Information
Published In
November 2023
237 pages
ISBN:9798400709272
DOI:10.1145/3638682
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 22 May 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- ?????????????????? YQJH
Conference
VSIP 2023
VSIP 2023: 2023 the 5th International Conference on Video, Signal and Image Processing
November 24 - 26, 2023
Harbin, China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 15Total Downloads
- Downloads (Last 12 months)15
- Downloads (Last 6 weeks)3
Reflects downloads up to 02 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format