Abstract
With the development of imaging technology, it becomes increasingly easy to obtain hyperspectral images (HSI) containing rich spectral information. The application of hyperspectral images in autonomous driving is expected to become a reality. Hyperspectral images have higher dimensions and more information than RGB images, which brings the challenge of training with limited samples. It is not easy to train satisfactory semantic segmentation models directly from hyperspectral images. In this paper, we propose a hyperspectral dual-fusion (hyperDF) network that can fully utilize the pre-training knowledge of the RGB modality to improve the performance of hyperspectral semantic segmentation. Specifically, we generate pseudo-color images from hyperspectral images to simulate RGB signals in order to make better use of the pre-trained RGB semantic segmentation models. The pseudo-color and hyperspectral images are processed in a dual encoder structure and then fused through a channel-wise attention fusion module. Finally, a multi-scale decoder injected with low-level features is used to predict the semantic segmentation results. Experimental results on the Hyperspectral City V2.0 dataset show that our method achieves state-of-the-art results, with a mIoU of 51.68%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Hyperspectral city challenge (2021). https://pbdl-ws.github.io/pbdl2021/challenge/index.html
Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), 2481–2495 (2017)
Berman, M., Triki, A.R., Blaschko, M.B.: The lovász-softmax loss: a tractable surrogate for the optimization of the intersection-over-union measure in neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4413–4421 (2018)
Cao, X., Tong, X., Dai, Q., Lin, S.: High resolution multispectral video capture with a hybrid camera system. In: CVPR 2011, pp. 297–304. IEEE (2011)
Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 (2017)
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 833–851. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_49
Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3213–3223 (2016)
Fu, J., et al.: Dual attention network for scene segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3146–3154 (2019)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)
Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., Liu, W.: CCNet: criss-cross attention for semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 603–612 (2019)
iori2333: Hyperspectral city v2 benchmark. https://github.com/iori2333/HSICityV2-Benchmark
Kemker, R., Kanan, C.: Self-taught feature learning for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 55(5), 2693–2705 (2017)
Li, S., Song, W., Fang, L., Chen, Y., Ghamisi, P., Benediktsson, J.A.: Deep learning for hyperspectral image classification: an overview. IEEE Trans. Geosci. Remote Sens. 57(9), 6690–6709 (2019)
Li, Z., Xiong, F., Zhou, J., Wang, J., Lu, J., Qian, Y.: Bae-net: a band attention aware ensemble network for hyperspectral object tracking. In: 2020 IEEE International Conference on Image Processing (ICIP), pp. 2106–2110. IEEE (2020)
Liu, Z., Wang, X., Zhong, Y., Shu, M., Sun, C.: Siamhyper: learning a hyperspectral object tracker from an RGB-based tracker. IEEE Trans. Image Process. 31, 7116–7129 (2022)
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)
Lu, J., Gong, P., Ye, J., Zhang, C.: Learning from very few samples: a survey. arXiv preprint arXiv:2009.02653 (2020)
Pan, B., Shi, Z., Xu, X., Shi, T., Zhang, N., Zhu, X.: CoinNet: copy initialization network for multispectral imagery semantic segmentation. IEEE Geosci. Remote Sens. Lett. 16(5), 816–820 (2018)
Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Sun, H., Zheng, X., Lu, X.: A supervised segmentation network for hyperspectral image classification. IEEE Trans. Image Process. 30, 2810–2825 (2021)
Sun, K., et al.: High-resolution representations for labeling pixels and regions. arXiv preprint arXiv:1904.04514 (2019)
Vunckx, K., Charle, W.: Accurate video-rate multi-spectral imaging using imec snapshot sensors. In: 2021 11th Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS), pp. 1–7. IEEE (2021)
Wang, C., Ye, M., Lei, L., Xiong, F., Qian, Y.: Cross-domain attention network for hyperspectral image classification. In: IGARSS 2022–2022 IEEE International Geoscience and Remote Sensing Symposium, pp. 1564–1567. IEEE (2022)
Wang, X., Tan, K., Du, Q., Chen, Y., Du, P.: Cva 2 e: a conditional variational autoencoder with an adversarial training process for hyperspectral imagery classification. IEEE Trans. Geosci. Remote Sens. 58(8), 5676–5692 (2020)
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: Segformer: simple and efficient design for semantic segmentation with transformers. Adv. Neural. Inf. Process. Syst. 34, 12077–12090 (2021)
Xu, C., et al.: Image2point: 3D point-cloud understanding with 2d image pretrained models. In: ECCV 2022, Part XXXVII. LNCS, vol. 13697, pp. 638–656. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19836-6_36
Yang, J., Zhao, Y.Q., Chan, J.C.W.: Learning and transferring deep joint spectral-spatial features for hyperspectral classification. IEEE Trans. Geosci. Remote Sens. 55(8), 4729–4742 (2017)
Zhang, P., Zhao, J., Wang, D., Lu, H., Ruan, X.: Visible-thermal UAV tracking: a large-scale benchmark and new baseline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8886–8895 (2022)
Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2881–2890 (2017)
Zhu, L., Chen, Y., Ghamisi, P., Benediktsson, J.A.: Generative adversarial networks for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 56(9), 5046–5063 (2018)
Acknowledgment
This work was supported in part by the National Natural Science Foundation of China under Grant 62106106.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ding, X., Gu, S., Yang, J. (2023). Dual Fusion Network for Hyperspectral Semantic Segmentation. In: Lu, H., et al. Image and Graphics. ICIG 2023. Lecture Notes in Computer Science, vol 14356. Springer, Cham. https://doi.org/10.1007/978-3-031-46308-2_13
Download citation
DOI: https://doi.org/10.1007/978-3-031-46308-2_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-46307-5
Online ISBN: 978-3-031-46308-2
eBook Packages: Computer ScienceComputer Science (R0)