Skip to main content
Springer Nature Link
Log in

Remote Sensing Image Fusion Based on Two-Stream Fusion Network

  • Conference paper
  • First Online:
MultiMedia Modeling (MMM 2018)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 10704))

Included in the following conference series:

Abstract

Remote sensing image fusion (or pan-sharpening) aims at generating high resolution multi-spectral (MS) image from inputs of a high spatial resolution single band panchromatic (PAN) image and a low spatial resolution multi-spectral image. In this paper, a deep convolutional neural network with two-stream inputs respectively for PAN and MS images is proposed for remote sensing image pan-sharpening. Firstly the network extracts features from PAN and MS images, then it fuses them to form compact feature maps that can represent both spatial and spectral information of PAN and MS images, simultaneously. Finally, the desired high spatial resolution MS image is recovered from the fused features using an encoding-decoding scheme. Experiments on Quickbird satellite images demonstrate that the proposed method can fuse the PAN and MS image effectively.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    w and h are the width and height of the input images.

  2. 2.

    http://www.digitalglobe.com/.

References

  1. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., et al.: TensorFlow: large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 (2016)

  2. Alparone, L., Baronti, S., Garzelli, A., Nencini, F.: A global quality measurement of pan-sharpened multispectral imagery. GRSL 1(4), 313–317 (2004)

    Google Scholar 

  3. Aly, H.A., Sharma, G.: A regularized model-based optimization framework for pan-sharpening. TIP 23(6), 2596–2608 (2014)

    MathSciNet  MATH  Google Scholar 

  4. Chavez, P., Sides, S.C., Anderson, J.A., et al.: Comparison of three different methods to merge multiresolution and multispectral data - landsat TM and SPOT panchromatic. PE&RS 57(3), 295–303 (1991)

    Google Scholar 

  5. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. TPAMI 38(2), 295–307 (2016)

    Article  Google Scholar 

  6. González-Audícana, M., Saleta, J.L., Catalán, R.G., García, R.: Fusion of multispectral and panchromatic images using improved IHS and PCA mergers based on wavelet decomposition. TGRS 42(6), 1291–1299 (2004)

    Google Scholar 

  7. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)

    Google Scholar 

  8. He, X., Condat, L., Bioucas-Dias, J.M., Chanussot, J., Xia, J.: A new pansharpening method based on spatial and spectral sparsity priors. TIP 23(9), 4160–4174 (2014)

    MathSciNet  MATH  Google Scholar 

  9. Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: CVPR, pp. 1646–1654 (2016)

    Google Scholar 

  10. Kingma, D., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  11. Laben, C.A., Brower, B.V.: Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. US Patent 6,011,875, 4 January 2000

    Google Scholar 

  12. Li, S., Yang, B.: A new pan-sharpening method using a compressed sensing technique. TGRS 49(2), 738–746 (2011)

    Google Scholar 

  13. Liu, Q., Wang, Y., Zhang, Z.: Pan-sharpening based on geometric clustered neighbor embedding. OE 53(9), 093109 (2014)

    Article  Google Scholar 

  14. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR, pp. 3431–3440 (2015)

    Google Scholar 

  15. Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: ICML, vol. 30 (2013)

    Google Scholar 

  16. Masi, G., Cozzolino, D., Verdoliva, L., Scarpa, G.: Pansharpening by convolutional neural networks. Remote Sens. 8(7), 594 (2016)

    Article  Google Scholar 

  17. Nunez, J., Otazu, X., Fors, O., Prades, A., Pala, V., Arbiol, R.: Multiresolution-based image fusion with additive wavelet decomposition. TGRS 37(3), 1204–1211 (1999)

    Google Scholar 

  18. Otazu, X., González-Audícana, M., Fors, O., Núñez, J.: Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods. TGRS 43(10), 2376–2385 (2005)

    Google Scholar 

  19. Pradhan, P.S., King, R.L., Younan, N.H., Holcomb, D.W.: Estimation of the number of decomposition levels for a wavelet-based multiresolution multisensor image fusion. TGRS 44(12), 3674–3686 (2006)

    Google Scholar 

  20. Prashanth, H., Shashidhara, H., Balasubramanya Murthy, K.N.: Image scaling comparison using universal image quality index. In: IAC3T, pp. 859–863. IEEE (2009)

    Google Scholar 

  21. Rahmani, S., Strait, M., Merkurjev, D., Moeller, M., Wittman, T.: An adaptive IHS pan-sharpening method. GRSL 7(4), 746–750 (2010)

    Google Scholar 

  22. Ranchin, T., Wald, L.: Fusion of high spatial and spectral resolution images: the arsis concept and its implementation. PE&RS 66(1), 49–61 (2000)

    Google Scholar 

  23. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: NIPS, pp. 91–99 (2015)

    Google Scholar 

  24. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  25. Shah, V.P., Younan, N.H., King, R.L.: An efficient pan-sharpening method via a combined adaptive PCA approach and contourlets. TGRS 46(5), 1323–1335 (2008)

    Google Scholar 

  26. Thomas, C., Ranchin, T., Wald, L., Chanussot, J.: Synthesis of multispectral images to high spatial resolution: a critical review of fusion methods based on remote sensing physics. TGRS 46(5), 1301–1312 (2008)

    Google Scholar 

  27. Tu, T.M., Su, S.C., Shyu, H.C., Huang, P.S.: A new look at IHS-like image fusion methods. Inf. Fusion 2(3), 177–186 (2001)

    Article  Google Scholar 

  28. Vivone, G., Alparone, L., Chanussot, J., Dalla Mura, M., Garzelli, A., Licciardi, G.A., Restaino, R., Wald, L.: A critical comparison among pansharpening algorithms. TGRS 53(5), 2565–2586 (2015)

    Google Scholar 

  29. Wald, L.: Quality of high resolution synthesised images: is there a simple criterion? In: Proceedings of Fusion of Earth Data: Merging Point Measurements, Raster Maps, and Remotely Sensed image (2000)

    Google Scholar 

  30. Wald, L., Ranchin, T., Mangolini, M.: Fusion of satellite images of different spatial resolutions: assessing the quality of resulting images. PE&RS 63, 691–699 (1997)

    Google Scholar 

  31. Wei, Q., Dobigeon, N., Tourneret, J.Y.: Bayesian fusion of multi-band images. J-STSP 9(6), 1117–1127 (2015)

    Google Scholar 

  32. Yuhas, R.H., Goetz, A.F.H., Boardman, J.: Discrimination among semi-arid landscape endmembers using the spectral angle mapper (SAM) algorithm. In: Proceedings of Summaries 3rd Annual JPL Airborne Geoscience Workshop, pp. 147–149, June 1992

    Google Scholar 

  33. Zhang, Q., Wang, Y., Liu, Q., Liu, X., Wang, W.: CNN based suburban building detection using monocular high resolution Google earth images. In: IGARSS, pp. 661–664. IEEE (2016)

    Google Scholar 

  34. Zhong, J., Yang, B., Huang, G., Zhong, F., Chen, Z.: Remote sensing image fusion with convolutional neural network. Sens. Imaging 17(1), 10 (2016)

    Article  Google Scholar 

  35. Zhu, X.X., Bamler, R.: A sparse image fusion algorithm with application to pan-sharpening. TGRS 51(5), 2827–2836 (2013)

    Google Scholar 

Download references

Acknowledgments

This work is supported by the Natural Science Foundation of China (NSFC) under Grant No. 61601011.

Author information

Authors and Affiliations

  1. The State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, 100191, China

    Xiangyu Liu, Yunhong Wang & Qingjie Liu

  2. Beijing Key Laboratory of Digital Media, School of Computer Science and Engineering, Beihang University, Beijing, 100191, China

    Qingjie Liu

Authors
  1. Xiangyu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yunhong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qingjie Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qingjie Liu .

Editor information

Editors and Affiliations

  1. Alpen-Adria-Universität Klagenfurt, Klagenfurt, Austria

    Klaus Schoeffmann

  2. Chulalongkorn University, Bangkok, Thailand

    Thanarat H. Chalidabhongse

  3. City University of Hong Kong, Hong Kong, China

    Chong Wah Ngo

  4. Chulalongkorn University, Bangkok, Thailand

    Supavadee Aramvith

  5. Dublin City University, Dublin, Ireland

    Noel E. O’Connor

  6. Gwangju Institute of Science and Technology, Gwangju, Korea (Republic of)

    Yo-Sung Ho

  7. Tampere University of Technology, Tampere, Finland

    Moncef Gabbouj

  8. Rutgers University, Piscataway, New Jersey, USA

    Ahmed Elgammal

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Liu, X., Wang, Y., Liu, Q. (2018). Remote Sensing Image Fusion Based on Two-Stream Fusion Network. In: Schoeffmann, K., et al. MultiMedia Modeling. MMM 2018. Lecture Notes in Computer Science(), vol 10704. Springer, Cham. https://doi.org/10.1007/978-3-319-73603-7_35

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-73603-7_35

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-73602-0

  • Online ISBN: 978-3-319-73603-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics