research-article

Pansharpening Scheme Using Bi-dimensional Empirical Mode Decomposition and Neural Network

Authors:

Balasubramanian RamanAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 18, Issue 4

Article No.: 108, Pages 1 - 22

https://doi.org/10.1145/3506709

Published: 04 March 2022 Publication History

Abstract

The pansharpening is a combination of multispectral (MS) and panchromatic (PAN) images that produce a high-spatial-spectral-resolution MS images. In multiresolution analysis–based pansharpening schemes, some spatial and spectral distortions are found. It can be reduced by adding spatial detail images of the PAN image into MS images. In the convolution neural network– (CNN) based method, the lowpass filter image extracted by the CNN model when MS and PAN images are directly applied into the input. The feature values are very high and reduce the conversion efficiency. In the proposed scheme, bi-dimensional empirical mode decomposition is used to extract the spatial detail information of the PAN image to reduce the feature values of the input. This extracted PAN image information is applied to the CNN to produce the non-linear changes in the image pixels and transformed into the perfect spatial detail image. It identifies the spatial and spectral detail quantity for the proposed scheme and it also varies with the different datasets automatically of the same satellite images. Simulation results in the context of qualitative and quantitative analysis demonstrate the effectiveness of proposed scheme applied on datasets collected by different satellites.

References

[1]

B. Aiazzi, L. Alparone, S. Baronti, A. Garzelli, and M. Selva. 2003. An MTF-based spectral distortion minimizing model for pan-sharpening of very high resolution multispectral images of urban areas. In Proceedings of the 2nd GRSS/ISPRS Joint Workshop on Remote Sensing and Data Fusion over Urban Areas. IEEE, 90–94.

[2]

B. Aiazzi, L. Alparone, S. Baronti, A. Garzelli, and M. Selva. 2006. MTF-tailored multiscale fusion of high-resolution MS and pan imagery. Photogram. Eng. Remote Sens. 72, 5 (2006), 591–596.

[3]

Bruno Aiazzi, Stefano Baronti, Massimo Selva, and Luciano Alparone. 2013. Bi-cubic interpolation for shift-free pan-sharpening. ISPRS J. Photogram. Remote Sensing 86 (2013), 65–76.

[4]

Luciano Alparone, Bruno Aiazzi, Stefano Baronti, Andrea Garzelli, Filippo Nencini, and Massimo Selva. 2008. Multispectral and panchromatic data fusion assessment without reference. Photogram. Eng. Remote Sens. 74, 2 (2008), 193–200.

[5]

Luciano Alparone, Lucien Wald, Jocelyn Chanussot, Claire Thomas, Paolo Gamba, and Lori Mann Bruce. 2007. Comparison of pansharpening algorithms: Outcome of the 2006 GRS-S data-fusion contest. IEEE Trans. Geosci. Remote Sens. 45, 10 (2007), 3012–3021.

[6]

Arian Azarang, Hafez E Manoochehri, and Nasser Kehtarnavaz. 2019. Convolutional autoencoder-based multispectral image fusion. IEEE Access 7 (2019), 35673–35683.

[7]

Pats Chavez, Stuart C. Sides, Jeffrey A. Anderson, et al. 1991. Comparison of three different methods to merge multiresolution and multispectral data- landsat TM and SPOT panchromatic. Photogram. Eng. Remote Sens. 57, 3 (1991), 295–303.

[8]

Wojciech Czaja, Timothy Doster, and James M. Murphy. 2014. Wavelet packet mixing for image fusion and pan-sharpening. In Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XX, Vol. 9088. International Society for Optics and Photonics, 908803.

[9]

Mauro Dalla Mura, Saurabh Prasad, Fabio Pacifici, Paulo Gamba, Jocelyn Chanussot, and Jón Atli Benediktsson. 2015. Challenges and opportunities of multimodality and data fusion in remote sensing. Proc. IEEE 103, 9 (2015), 1585–1601.

[10]

Weihua Dong, Xian’en Li, Xiangguo Lin, Zhilin Li, et al. 2014. A bidimensional empirical mode decomposition method for fusion of multispectral and panchromatic remote sensing images. Remote Sens. 6, 9 (2014), 8446–8467.

[11]

Patrick Flandrin, Gabriel Rilling, and Paulo Goncalves. 2004. Empirical mode decomposition as a filter bank. IEEE Sign. Process. Lett. 11, 2 (2004), 112–114.

[12]

Alan R. Gillespie, Anne B. Kahle, and Richard E. Walker. 1987. Color enhancement of highly correlated images. II. Channel ratio and “chromaticity” transformation techniques. Remote Sens’ Environ’ 22, 3 (1987), 343–365.

[13]

María González-Audícana, José Luis Saleta, Raquel García Catalán, and Rafael García. 2004. Fusion of multispectral and panchromatic images using improved IHS and PCA mergers based on wavelet decomposition. IEEE Trans’ Geosci. Remote Sens. 42, 6 (2004), 1291–1299.

[14]

Lin He, Yizhou Rao, Jun Li, Jocelyn Chanussot, Antonio Plaza, Jiawei Zhu, and Bo Li. 2019. Pansharpening via detail injection based convolutional neural networks. IEEE J. Select. Top. Appl. Earth Observ. Remote Sens. (2019).

[15]

A. K. Helmy and Gh S. El-Tawel. 2015. An integrated scheme to improve pan-sharpening visual quality of satellite images. Egypt. Inf. J. 16, 1 (2015), 121–131.

[16]

Norden E. Huang, Zheng Shen, Steven R. Long, Manli C. Wu, Hsing H. Shih, Quanan Zheng, Nai-Chyuan Yen, Chi Chao Tung, and Henry H. Liu. 1998. The empirical mode decomposition and the hilbert spectrum for nonlinear and non-stationary time series analysis. In Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, Vol. 454. The Royal Society, 903–995.

[17]

Manjunath Joshi and André Jalobeanu. 2009. MAP estimation for multiresolution fusion in remotely sensed images using an IGMRF prior model. IEEE Trans. Geosci. Remote Sens. 48, 3 (2009), 1245–1255.

[18]

Xudong Kang, Shutao Li, and Jón Atli Benediktsson. 2013. Pansharpening with matting model. IEEE Trans. Geosci. Remote Sens. 52, 8 (2013), 5088–5099.

[19]

Hossam M. Kasem, Kwok-Wai Hung, and Jianmin Jiang. 2018. Revised spatial transformer network towards improved image super-resolutions. In Proceedings of the 24th International Conference on Pattern Recognition (ICPR’18). IEEE, 2688–2692.

[20]

Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1646–1654.

[21]

Jonghwa Lee and Chulhee Lee. 2010. Fast and efficient panchromatic sharpening. IEEE Trans. Geosci. Remote Sens. 48, 1 (2010), 155–163.

[22]

Shutao Li and Bin Yang. 2010. A new pan-sharpening method using a compressed sensing technique. IEEE Trans. Geosci. Remote Sens. 49, 2 (2010), 738–746.

[23]

Shutao Li, Haitao Yin, and Leyuan Fang. 2013. Remote sensing image fusion via sparse representations over learned dictionaries. IEEE Trans. Geosci. Remote Sens. 51, 9 (2013), 4779–4789.

[24]

Yanping Li and Shengming Jiang. 2020. Multi-focus image fusion using geometric algebra based discrete fourier transform. IEEE Access 8 (2020), 60019–60028.

[25]

Giuseppe Masi, Davide Cozzolino, Luisa Verdoliva, and Giuseppe Scarpa. 2016. Pansharpening by convolutional neural networks. Remote Sens. 8, 7 (2016), 594.

[26]

Jean Claude Nunes, Yasmina Bouaoune, Eric Delechelle, Oumar Niang, and Ph Bunel. 2003. Image analysis by bidimensional empirical mode decomposition. Image Vis. Comput. 21, 12 (2003), l019–1026.

[27]

Xavier Otazu, María González-Audícana, Octavi Fors, and Jorge Núñez. 2005. Introduction of sensor spectral response into image fusion methods. application to wavelet-based methods. IEEE Trans. Geosci. Remote Sens. 43, 10 (2005), 2376–2385.

[28]

Rocco Restaino, Gemine Vivone, Mauro Dalla Mura, and Jocelyn Chanussot. 2016. Fusion of multispectral and panchromatic images based on morphological operators. IEEE Trans. Image Process. 25, 6 (2016), 2882–2895.

[29]

Nidhi Saxena and Raman Balasubramanian. 2021. A pansharpening scheme using spectral graph wavelet transforms and convolutional neural networks. Int. J. Remote Sens. 42, 8 (2021), 2898–2919.

[30]

Nidhi Saxena and Gaurav Saxena. 2021. Pansharpening scheme using spatial detail injection based convolutional neural networks. IET Image Process. (2021).

[31]

Nidhi Saxena and K. K. Sharma. 2017. A hybrid approach for pansharpening using hilbert vibration decomposition. In Proceedings of the IEEE International Conference on Power, Control, Signals and Instrumentation Engineering (ICPCSI’17). IEEE, 2801–2805.

[32]

Nidhi Saxena and K. K. Sharma. 2019. A two-dimensional discrete fractional fourier transform-based pansharpening scheme. Int. J. Remote Sens. 40, 16 (2019), 6098–6115.

[33]

Nidhi Saxena and Kamalesh K. Sharma. 2017. Pansharpening approach using hilbert vibration decomposition. IET Image Process. 11, 12 (2017), 1152–1162.

[34]

Nidhi Saxena and Kamalesh K. Sharma. 2018. Pansharpening scheme using filtering in two-dimensional discrete fractional fourier transform. IET Image Process. 12, 6 (2018), 1013–1019.

[35]

Giuseppe Scarpa, Sergio Vitale, and Davide Cozzolino. 2018. Target-adaptive CNN-based pansharpening. IEEE Trans. Geosci. Remote Sens. 56, 9 (2018), 5443–5457.

[36]

Vijay P. Shah, Nicolas H. Younan, and Roger L. King. 2008. An efficient pan-sharpening method via a combined adaptive PCA approach and contourlets. IEEE Trans. Geosci. Remote Sens. 46, 5 (2008), 1323–1335.

[37]

Huihui Song, Bo Huang, Qingshan Liu, and Kaihua Zhang. 2015. Improving the spatial resolution of landsat TM/ETM+ through fusion with SPOT5 images via learning-based super-resolution. IEEE Trans. Geosci. Remote Sens. 53, 3 (2015), 1195–1204.

[38]

D. Stroppiana, K. Tansey, J. Gregoire, and J. M. C. Pereira. 2003. An algorithm for mapping burnt areas in Australia using SPOT-VEGETATION data. IEEE Trans. Geosci. Remote Sens. 41, 4 (April 2003), 907–909. DOI:

[39]

Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. 2018. Scale-recurrent network for deep image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8174–8182.

[40]

Claire Thomas, Thierry Ranchin, Lucien Wald, and Jocelyn Chanussot. 2008. Synthesis of multispectral images to high spatial resolution: A critical review of fusion methods based on remote sensing physics. IEEE Trans. Geosci. Remote Sens. 46, 5 (2008), 1301–1312.

[41]

Chunwei Tian, Yong Xu, Lunke Fei, and Ke Yan. 2018. Deep learning for image denoising: A survey. In International Conference on Genetic and Evolutionary Computing. Springer, 563–572.

[42]

Te-Ming Tu, Ping Sheng Huang, Chung-Ling Hung, and Chien-Ping Chang. 2004. A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOS imagery. IEEE Geosci. Remote Sens. Lett. 1, 4 (2004), 309–312.

[43]

Sergio Vitale and Giuseppe Scarpa. 2020. A detail-preserving cross-scale learning strategy for CNN-based pansharpening. Remote Sens. 12, 3 (2020), 348.

[44]

Gemine Vivone, Luciano Alparone, Jocelyn Chanussot, Mauro Dalla Mura, Andrea Garzelli, Giorgio A Licciardi, Rocco Restaino, and Lucien Wald. 2015. A critical comparison among pansharpening algorithms. IEEE Trans. Geosci. Remote Sens. 53, 5 (2015), 2565–2586.

[45]

Gemine Vivone, Rocco Restaino, and Jocelyn Chanussot. 2018. Full scale regression-based injection coefficients for panchromatic sharpening. IEEE Trans. Image Process. 27, 7 (2018), 3418–3431.

[46]

Gemine Vivone, Rocco Restaino, Mauro Dalla Mura, Giorgio Licciardi, and Jocelyn Chanussot. 2014. Contrast and error-based fusion schemes for multispectral image pansharpening. IEEE Geosci. Remote Sens. Lett. 11, 5 (2014), 930–934.

[47]

Lucien Wald. 2002. Data Fusion: Definitions and Architectures: Fusion of Images of Different Spatial Resolutions. Presses des MINES.

[48]

Zhou Wang and Alan C. Bovik. 2002. A universal image quality index. IEEE Sign. Process. Lett. 9, 3 (2002), 81–84.

[49]

Yancong Wei, Qiangqiang Yuan, Huanfeng Shen, and Liangpei Zhang. 2017. Boosting the accuracy of multispectral image pansharpening by learning a deep residual network. IEEE Geosci. Remote Sens. Lett. 14, 10 (2017), 1795–1799.

[50]

Junyuan Xie, Linli Xu, and Enhong Chen. 2012. Image denoising and inpainting with deep neural networks. In Advances in Neural Information Processing Systems. 341–349.

Digital Library

[51]

Roberta H. Yuhas, Alexander F. H. Goetz, and Joe W. Boardman. 1992. Discrimination among semi-arid landscape endmembers using the spectral angle mapper (SAM) algorithm. In Proc. Summaries 3rd Annu. JPL Airborne Geosci. Workshop, vol. 1. 147–149.

[52]

Zhan-Li Sun, De-Shuang Huang, Yiu-Ming Cheung, Jiming Liu, and Guang-Bin Huang. 2005. Using FCMC, FVS, and PCA techniques for feature extraction of multispectral images. IEEE Geosci. Remote Sens. Lett. 2, 2 (April 2005), 108–112. DOI:

[53]

Liangpei Zhang, Huanfeng Shen, Wei Gong, and Hongyan Zhang. 2012. Adjustable model-based fusion method for multispectral and panchromatic images. IEEE Trans. Syst. Man Cybernet. B 42, 6 (2012), 1693–1704.

Digital Library

[54]

W. Zhou. 2013. An object-based approach for urban land cover classification: Integrating LiDAR height and intensity data. IEEE Geosci. Remote Sens. Lett. 10, 4 (July 2013), 928–931. DOI:

[55]

Xiao Xiang Zhu and Richard Bamler. 2012. A sparse image fusion algorithm with application to pan-sharpening. IEEE Trans. Geosci. Remote Sens. 51, 5 (2012), 2827–2836.

[56]

H. Zhuang, K. Deng, H. Fan, and M. Yu. 2016. Strategies combining spectral angle mapper and change vector analysis to unsupervised change detection in multispectral images. IEEE Geosci. Remote Sens. Lett. 13, 5 (May 2016), 681–685. DOI:

Cited By

Sangani DThakker RPanchal SGogineni R(2023)Remote Sensing Pansharpening with TV-H−1 Decomposition and PSO-Based Adaptive Weighting MethodInternational Journal of Image and Graphics10.1142/S021946782450061X25:01Online publication date: 25-Jul-2023
https://doi.org/10.1142/S021946782450061X
Xu QLi GYang Y(2023)Application of Energy-Saving Technology for Carbon Dioxide Transcritical Wide Temperature Range Based on EMD Algorithm2023 International Conference on Data Science and Network Security (ICDSNS)10.1109/ICDSNS58469.2023.10244784(1-7)Online publication date: 28-Jul-2023
https://doi.org/10.1109/ICDSNS58469.2023.10244784

Index Terms

Pansharpening Scheme Using Bi-dimensional Empirical Mode Decomposition and Neural Network
1. Computing methodologies
  1. Modeling and simulation
    1. Simulation types and techniques
      1. Multiscale systems

Recommendations

Video super resolution using convolutional neural network and image fusion techniques

Recently, huge progress has been achieved in the field of single image super resolution which augments the resolution of images. The idea behind super resolution is to convert low-resolution images into high-resolution images. SRCNN (Single Resolution ...
Compression of multispectral images with inverse pyramid decomposition
KES'10: Proceedings of the 14th international conference on Knowledge-based and intelligent information and engineering systems: Part III

In the paper is presented a new method for compression of multispectral images, based on the Inverse Difference Pyramid decomposition. The method is applicable for any number of multispectral images of same object. The processing is performed as ...
High-Quality Bayesian Pansharpening

Pansharpening is a process of acquiring a multi-spectral image with high spatial resolution by fusing a low resolution multi-spectral image with a corresponding high resolution panchromatic image. In this paper, a new pansharpening method based on the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 18, Issue 4

November 2022

497 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3514185

Editor:
Abdulmotaleb El Saddik
Mohamed Bin Zayed University of Artificial Intelligence, UAE and University of Ottawa, Canada

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 March 2022

Accepted: 01 December 2021

Revised: 01 December 2021

Received: 01 June 2021

Published in TOMM Volume 18, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
230
Total Downloads

Downloads (Last 12 months)23
Downloads (Last 6 weeks)4

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sangani DThakker RPanchal SGogineni R(2023)Remote Sensing Pansharpening with TV-H−1 Decomposition and PSO-Based Adaptive Weighting MethodInternational Journal of Image and Graphics10.1142/S021946782450061X25:01Online publication date: 25-Jul-2023
https://doi.org/10.1142/S021946782450061X
Xu QLi GYang Y(2023)Application of Energy-Saving Technology for Carbon Dioxide Transcritical Wide Temperature Range Based on EMD Algorithm2023 International Conference on Data Science and Network Security (ICDSNS)10.1109/ICDSNS58469.2023.10244784(1-7)Online publication date: 28-Jul-2023
https://doi.org/10.1109/ICDSNS58469.2023.10244784

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents