Skip to main content
SpringerLink
Log in

V3O2: hybrid deep learning model for hyperspectral image classification using vanilla-3D and octave-2D convolution

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Remote sensing image analysis is an emerging area of research and is used for various applications such as climate analysis, crop monitoring and change detection. Hyperspectral image (HSI) is one of the dominant remote sensing imaging modalities that captures information beyond the visible spectrum. The evolution of deep learning has made a significant impact on HSI analysis, mainly for its classification. The spatial–spectral feature-based classification model improves the classification accuracy of hyperspectral images (HSIs). However, these models are computationally expensive, and redundancy exists in the spatial dimension of features. This research work proposes a hybrid convolutional neural network (CNN) for HSI classification. The proposed model uses principal component analysis (PCA) as a preprocessing technique for optimal band extraction from HSIs. The hybrid CNN classification technique extracts the spectral and spatial features using three-dimensional CNN (3D CNN). These features are fed into a two-dimensional CNN (2D CNN) for further feature extraction and classification. The redundancy in spatial features of the hybrid CNN model is reduced by octave convolution (OctConv) instead of standard vanilla convolution. OctConv factorizes the spatial features into lower and higher spatial frequencies, and different convolutions are performed on them based on their frequencies. The hybrid model is compared against various state-of-the-art CNN-based techniques and found that the accuracy is boosted with a lesser computational cost.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Camps-Valls, G., Gómez-Chova, L., Muñoz, J., Vila-Francés, J., Calpe, J.: Composite kernels for hyperspectral image classification. Geosci. Remote Sens. Lett. IEEE 3, 93–97 (2006). https://doi.org/10.1109/LGRS.2005.857031

    Article  Google Scholar 

  2. Chen, C., Jiang, F., Yang, C., Rho, S., Shen, W., Liu, S., Liu, Z.: Hyperspectral classification based on spectral–spatial convolutional neural networks. Eng. Appl. Artif. Intell. 68, 165–171 (2018). https://doi.org/10.1016/j.engappai.2017.10.015

    Article  Google Scholar 

  3. Chen, Y., Fang, H., Xu, B., Yan, Z., Kalantidis, Y., Rohrbach, M., Yan, S., Feng, J.: Drop an octave: reducing spatial redundancy in convolutional neural networks with octave convolution. arXiv preprint. arXiv:1904.05049 (2019)

  4. Chen, Y., Jiang, H., Li, C., Jia, X., Ghamisi, P.: Deep feature extraction and classification of hyperspectral images based on convolutional neural networks. IEEE Trans. Geosci. Remote Sens. 54(10), 6232–6251 (2016). https://doi.org/10.1109/TGRS.2016.2584107

    Article  Google Scholar 

  5. Chen, Y., Lin, Z., Zhao, X., Member, S., Wang, G., Gu, Y.: Deep learning-based classification of hyperspectral data. IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens. 7(6), 2094–2107 (2014). https://doi.org/10.1109/JSTARS.2014.2329330

    Article  Google Scholar 

  6. Congalton, R., Green, K.: Assessing the Accuracy of Remotely Sensed Data: Principles and Practices, 3rd edn. CRC Press, Boca Raton (2019). https://doi.org/10.1201/9780429052729

    Book  Google Scholar 

  7. Fauvel, M., Chanussot, J., Benediktsson, J.A.: A spatial-spectral kernel-based approach for the classification of remote-sensing images. Pattern Recogn. 45(1), 381–392 (2012). https://doi.org/10.1016/j.patcog.2011.03.035

    Article  Google Scholar 

  8. Gao, H., Yao, D., Wang, M., Li, C., Liu, H., Hua, Z., Wang, J.: A hyperspectral image classification method based on multi-discriminator generative adversarial networks. Sensors 19(15), 3269 (2019). https://doi.org/10.3390/s19153269

    Article  Google Scholar 

  9. Ham, J., Chen, Yangchi, Crawford, M.M., Ghosh, J.: Investigation of the random forest framework for classification of hyperspectral data. IEEE Trans Geosci Remote Sens 43(3), 492–501 (2005). https://doi.org/10.1109/TGRS.2004.842481

    Article  Google Scholar 

  10. Hang, R., Liu, Q., Hong, D., Ghamisi, P.: Cascaded recurrent neural networks for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 57(8), 1 (2019). https://doi.org/10.1109/TGRS.2019.2899129

    Article  Google Scholar 

  11. Haut, J.M., Paoletti, M.E., Plaza, J., Li, J., Plaza, A.: Active learning with convolutional neural networks for hyperspectral image classification using a new bayesian approach. IEEE Trans. Geosci. Remote Sens. 56(11), 6440–6461 (2018). https://doi.org/10.1109/TGRS.2018.2838665

    Article  Google Scholar 

  12. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 770–778 (2015)

  13. Hu, W.S., Li, H.C., Member, S., Pan, L., Li, W., Member, S.: Feature Extraction and Classification Based on Spatial-Spectral ConvLSTM Neural Network for Hyperspectral Images. In: IEEE Transactions on Geoscience and Remote Sensing, pp. 1–15 (2019)

  14. Jensen, J.: Remote Sensing of the Environment: An Earth Resource Perspective, 2nd edn. Prentice Hall, Upper Saddle River (2007)

    Google Scholar 

  15. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, vol. 25, pp. 1097–1105. Curran Associates, Inc., New York (2012)

  16. Lee, H., Kwon, H.: Going deeper with contextual CNN for hyperspectral image classification. IEEE Trans. Image Process. 26(10), 4843–4855 (2017). https://doi.org/10.1109/TIP.2017.2725580

    Article  MathSciNet  Google Scholar 

  17. Li, J., Xi, B., Li, Y., Du, Q., Wang, K.: Hyperspectral classification based on texture feature enhancement and deep belief networks. Remote Sens. 10(3), 396 (2018). https://doi.org/10.3390/rs10030396

    Article  Google Scholar 

  18. Ma, A., Filippi, A.M., Wang, Z., Yin, Z.: Hyperspectral image classification using similarity measurements-based deep recurrent neural networks. Remote Sens. 11(2), 1–19 (2019). https://doi.org/10.3390/rs11020194

    Article  Google Scholar 

  19. Ma, X., Fu, A., Wang, J., Wang, H., Yin, B.: Hyperspectral image classification based on deep deconvolution network with skip architecture. IEEE Trans. Geosci. Remote Sens. 56(8), 4781–4791 (2018). https://doi.org/10.1109/TGRS.2018.2837142

    Article  Google Scholar 

  20. Ma, X., Geng, J., Wang, H.: Hyperspectral image classification via contextual deep learning. Eurasip J. Image Video Process. 1, 20 (2015). https://doi.org/10.1186/s13640-015-0071-8

    Article  Google Scholar 

  21. Mei, S., Ji, J., Geng, Y., Zhang, Z., Li, X., Du, Q.: Unsupervised spatial-spectral feature learning by 3D convolutional autoencoder for hyperspectral classification. IEEE Trans. Geosci. Remote Sens. 57(9), 6808–6820 (2019). https://doi.org/10.1109/TGRS.2019.2908756

    Article  Google Scholar 

  22. Melgani, F., Bruzzone, L.: Classification of hyperspectral remote sensing images with support vector machines. IEEE Trans. Geosci. Remote Sens. 42, 1778–1790 (2004). https://doi.org/10.1109/TGRS.2004.831865

    Article  Google Scholar 

  23. Mughees, A., Ali, A., Tao, L.: Hyperspectral image classification via shape-adaptive deep learning. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 375–379 (2017)

  24. Pan, B., Shi, Z., Xu, X.: MugNet: deep learning for hyperspectral image classification using limited samples. ISPRS J. Photogramm. Remote Sens. (2017). https://doi.org/10.1016/j.isprsjprs.2017.11.003

    Article  Google Scholar 

  25. Paoletti, M.E., Haut, J.M., Fernandez-Beltran, R., Plaza, J., Plaza, A., Li, J., Pla, F.: Capsule networks for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 57(4), 2145–2160 (2019). https://doi.org/10.1109/TGRS.2018.2871782

    Article  Google Scholar 

  26. Paoletti, M.E., Haut, J.M., Fernandez-Beltran, R., Plaza, J., Plaza, A.J., Pla, F.: Deep pyramidal residual networks for spectral-spatial hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 57(2), 740–754 (2019). https://doi.org/10.1109/TGRS.2018.2860125

    Article  Google Scholar 

  27. Rodarmel, C., Shan, J.: Principal component analysis for hyperspectral image classification. Surv. Land Inf. Syst. 62, 115–122 (2002)

    Google Scholar 

  28. Roy, S.K., Krishna, G., Dubey, S.R., B Chaudhuri, B.: HybridSN : Exploring 3D-2D CNN feature hierarchy for hyperspectral image classification. In: IEEE Geoscience and Remote Sensing Letters (2019)

  29. Shi, C., Pun, C.M.: Multi-scale hierarchical recurrent neural networks for hyperspectral image classification. Neurocomputing 294, 82–93 (2018). https://doi.org/10.1016/j.neucom.2018.03.012

    Article  Google Scholar 

  30. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR. arXiv:1409.1556 (2014)

  31. Tarabalka, Y., Fauvel, M., Chanussot, J., Benediktsson, J.: Svm- and mrf-based method for accurate classification of hyperspectral images. Geosci. Remote Sens. Lett. IEEE 7, 736–740 (2010). https://doi.org/10.1109/LGRS.2010.2047711

    Article  Google Scholar 

  32. Xu, Y., Member, S., Zhang, L., Member, S., Du, B., Member, S., Zhang, F.: Spectral–spatial unified networks for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 56(10), 5893–5909 (2018). https://doi.org/10.1109/TGRS.2018.2827407

    Article  Google Scholar 

  33. Yu, S., Jia, S., Xu, C.: Convolutional neural networks for hyperspectral image classification. Neurocomputing 219, 88–98 (2017). https://doi.org/10.1016/j.neucom.2016.09.010

    Article  Google Scholar 

  34. Zhong, Z., Li, J., Clausi, D.A., Wong, A.: Generative adversarial networks and conditional random fields for hyperspectral image classification. In: IEEE Transactions on Cybernetics, pp. 1–12 (2019). https://doi.org/10.1109/tcyb.2019.2915094

  35. Zhong, Z., Li, J., Luo, Z., Chapman, M.: Spectral–spatial residual network for hyperspectral image classification: a 3-d deep learning framework. IEEE Trans. Geosci. Remote Sens. 56(2), 847–858 (2018). https://doi.org/10.1109/TGRS.2017.2755542

    Article  Google Scholar 

  36. Zhou, F., Hang, R., Liu, Q., Yuan, X.: Hyperspectral image classification using spectral-spatial LSTMs. Neurocomputing 328, 39–47 (2019). https://doi.org/10.1016/j.neucom.2018.02.105

    Article  Google Scholar 

  37. Zhu, J., Hu, J., Jia, S., Jia, X., Li, Q.: Multiple 3-D feature fusion framework for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 56(4), 1873–1886 (2018). https://doi.org/10.1109/TGRS.2017.2769113

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, National Institute of Technology Karnataka, Surathkal, Mangalore, Karnataka, 575025, India

    Alkha Mohan & Venkatesan Meenakshi Sundaram

Authors
  1. Alkha Mohan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Venkatesan Meenakshi Sundaram
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alkha Mohan.

Ethics declarations

Conflict of interest

The authors declare no potential conflict of interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mohan, A., Meenakshi Sundaram, V. V3O2: hybrid deep learning model for hyperspectral image classification using vanilla-3D and octave-2D convolution. J Real-Time Image Proc 18, 1681–1695 (2021). https://doi.org/10.1007/s11554-020-00966-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-020-00966-z

Keywords

Search

Navigation