Skip to main content
SpringerLink
Log in

Panchromatic and multi-spectral image fusion for new satellites based on multi-channel deep model

  • Special Issue Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

With the launch and rapid development of new satellites such as WorldView-3, the bands number of multi-spectral images from new satellites is greatly increased. However, the spectral matching between the panchromatic image and multi-spectral images is deteriorated with the existing image fusion methods. In this paper, a novel method based on the multi-channel deep model is proposed to fuse images for new satellites. The deep model is implemented by convolutional neural networks and trained on each band to reduce the impact of spectral range mismatch. The proposed method also preserves the detailed information in multi-spectral images, which is ignored by the traditional methods. It also effectively alleviates the inconvenience for obtaining the remote sensing images by the data augmentation processing. In addition, it reduces the randomness of manual setting parameters using the parameter self-learning. Visual and quantitative assessments of fusion results show that the proposed method clearly improves the fusion quality compared to the state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Vivone, G., Alparone, L., Chanussot, J., Dalla Mura, M., Garzelli, A., Licciardi, G.A., Restaino, R., Wald, L.: A critical comparison among pansharpening algorithms. IEEE Trans. Geosci. Remote Sens. 53(5), 2565–2586 (2015)

    Article  Google Scholar 

  2. Li, S., Kang, X., Fang, L., Hu, J., Yin, H.: Pixel-level image fusion: a survey of the state of the art. Inf. Fusion 33, 100–112 (2017)

    Article  Google Scholar 

  3. Cetin, M., Tepecik, A.: Intensity–hue–saturation-based image fusion using iterative linear regression. J. Appl. Remote Sens. 10(4), 045019 (2016)

    Article  Google Scholar 

  4. Licciardi, G., Vivone, G., Dalla Mura, M., Restaino, R., Chanussot, J.: Multi-resolution analysis techniques and nonlinear PCA for hybrid pansharpening applications. Multidimens. Syst. Signal Process. 27(4), 807–830 (2016)

    Article  MathSciNet  Google Scholar 

  5. Guo, Q., Ehlers, M., Wang, Q., Pohl, C., Hornberg, S., Li, A.: Ehlers pan-sharpening performance enhancement using HCS transform for n-band data sets. Int. J. Remote Sens. 38(17), 4974–5002 (2017)

    Article  Google Scholar 

  6. Wang, W., Jiao, L., Yang, S.: Novel adaptive component-substitution-based pan-sharpening using particle swarm optimization. IEEE Geosci. Remote Sens. Lett. 12(4), 781–785 (2014)

    Article  Google Scholar 

  7. Wang, W., Li, H., Zhang, X.: Fusion algorithm of remote sensing images based on nonsubsampled pyramid and empirical mode demoposition. J. Harbin Eng. Univ. 11, 012 (2012)

    Google Scholar 

  8. Vishnu Pradeep, V., Sowmya, V., Soman, K.P.: Application of M-band wavelet in pan-sharpening. J. Intell. Fuzzy Syst. 32, 3151–3158 (2017)

    Article  Google Scholar 

  9. Chen, N., Niu, W., Zhang, J., Wang, K., Dai, E., Han, P.: Remote sensing image fusion algorithm based on modified contourlet transform. Electron. Des. Eng. 6, 58–61 (2017)

    Google Scholar 

  10. Otazu, X., González-Audácana, M., Fors, O., Núñez, J.: Introduction of sensor spectral response into image fusion method: application to wavelet-based methods. IEEE Trans. Geosci. Remote Sens. 43(10), 2376–2385 (2005)

    Article  Google Scholar 

  11. Garzelli, A., Nencini, F.: Interband structure modeling for pan-sharpening of very high-resolution multispectral images. Inf. Fusion 6(3), 213–224 (2005)

    Article  Google Scholar 

  12. Li, S., Yang, B.: A new pan-sharpening method using a compressed sensing technique. IEEE Trans. Geosci. Remote Sens. 49(2), 738–746 (2011)

    Article  Google Scholar 

  13. Jiang, C., Zhang, H., Shen, H., Zhang, L.: A practical compressed sensing-based pan-sharpening method. IEEE Geosci. Remote Sens. Lett. 9(4), 629–633 (2012)

    Article  Google Scholar 

  14. Longbotham, N., Pacifici, F., Malitz, S., Baugh, W., Camps-Valls, G.: Measuring the spatial and spectral performance of WorldView-3. In: Hyperspectral Imaging & Sounding of the Environment (2015). https://doi.org/10.1364/HISE.2015.HW3B.2

  15. Sibanda, M., Mutanga, O., Rouget, M.: Testing the capabilities of the new WorldView-3 space-borne sensor’s red-edge spectral band in discriminating and mapping complex grassland management treatments. Int. J. Remote Sens. 38(1), 1–22 (2017)

    Article  Google Scholar 

  16. Gong, Y., Wang, L., Guo, R., Lazebnik, S.: Multi-scale orderless pooling of deep convolutional activation features. Comput. Vision ECCV 8695, 392–407 (2014)

    Google Scholar 

  17. Yu, Z., Wu, F., Zhang, Y., Tang, S., Shao, J., Zhuang, Y.: Hashing with list-wise learning to rank. In: International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 999–1002 (2014)

  18. He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(9), 1904–1916 (2015)

    Article  Google Scholar 

  19. Ding, J., Chen, B., Liu, H., Huang, M.: Convolutional neural networks with data augmentation for SAR target recognition. IEEE Geosci. Remote Sens. Lett. 13(3), 364–368 (2016)

    Google Scholar 

  20. Xia, Z., Feng, X., Lin, J., Hadid, A.: Deep convolutional hashing using pairwise multi-label supervision for large-scale visual search. Signal Process. Image Commun. 59, 109–116 (2017)

    Article  Google Scholar 

  21. Huang, W., Xiao, L., Wei, Z., Liu, H., Tang, S.: A new pan-sharpening method with deep neural networks. IEEE Geosci. Remote Sens. Lett. 12(5), 1037–1041 (2017)

    Article  Google Scholar 

  22. Zhong, J., Yang, B., Huang, G., Zhong, F., Chen, Z.: Remote sensing image fusion with convolutional neural networks. Sens. Imaging 17(1), 10 (2016)

    Article  Google Scholar 

  23. Masi, G., Cozzolino, D., Verdoliva, L., Scarpa, G.: Pan-sharpening by convolutional neural networks. Remote Sens. 8(7), 594 (2016)

    Article  Google Scholar 

  24. Wang, H., Chen, S., Xu, F., Jin, Y.: Application of deep-learning algorithms to MSTAR data. In: Geoscience & Remote Sensing Symposium, pp. 3743–3745 (2015)

  25. Castelluccio, M., Poggi, G., Sansone, C., Verdoliva, L.: Land use classification in remote sensing images by convolutional neural networks. Acta Ecol. Sin. 28(2), 627–635 (2015)

    Google Scholar 

  26. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Computer Vision—ECCV 2014, vol. 8692, pp. 184–199 (2014)

  27. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295 (2014)

    Article  Google Scholar 

  28. Yu, Z., Yu, J., Fan, J., Tao, D.: Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. International conference on computer vision (ICCV), Venice, Italy (2017)

  29. Aiazzi, B., Alparone, L., Baronti, S., Garzelli, A.: Context-driven fusion of high spatial and spectral resolution images based on oversampled multi-resolution analysis. IEEE Trans. Geosci. Remote Sens. 40(10), 2300–2312 (2002)

    Article  Google Scholar 

  30. Chang, H., Yang, H., Gan, Y., Wang, M.: Sparse feature fidelity for perceptual image quality assessment. IEEE Trans. Image Process. Publ. IEEE Signal Process. Soc. 22(10), 4007–4018 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  31. Pushparaj, J., Hegde, A.V.: Comparison of various pan-sharpening methods using Quickbird-2 and Landsat-8 imagery. Arab. J. Geosci. 10(5), 119 (2017)

    Article  Google Scholar 

  32. Liu, P., Xiao, L., Tang, S.: A new geometry enforcing variational model for pan-sharpening. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 9(12), 5726–5739 (2016)

    Article  Google Scholar 

  33. Sanli, F.B., Abdikan, S., Esetlili, M.T., Sunar, F.: Evaluation of image fusion methods using PALSAR, RADARSAT-1 and SPOT images for land use/ land cover classification. J. Indian Soc. Remote Sens. 45(4), 591–601 (2017)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Nature Science Foundation of China (Nos. 61402368 and 61702419), Aerospace Support Fund, Fundamental Research Fund for the Central Universities.

Author information

Authors and Affiliations

  1. School of Electronics and Information, Northwestern Polytechnical University, Shaanxi, 710129, China

    Guiqing He, Siyuan Xing & Zhaoqiang Xia

  2. Institute of Remote Sensing and Digital Earth, Chinese Academy of Sciences, Beijing, 100101, China

    Qingqing Huang

  3. Department of Computer Science, University of North Carolina at Charlotte, Charlotte, NC, 28223, USA

    Jianping Fan

Authors
  1. Guiqing He
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Siyuan Xing
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhaoqiang Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qingqing Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jianping Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guiqing He.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

He, G., Xing, S., Xia, Z. et al. Panchromatic and multi-spectral image fusion for new satellites based on multi-channel deep model. Machine Vision and Applications 29, 933–946 (2018). https://doi.org/10.1007/s00138-018-0964-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-018-0964-5

Keywords

Search

Navigation