skip to main content
10.1145/3503161.3547774acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Normalization-based Feature Selection and Restitution for Pan-sharpening

Published: 10 October 2022 Publication History

Abstract

Pan-sharpening is essentially a panchromatic (PAN) image-guided low-spatial resolution MS image super-resolution problem. The commonly challenging issue of pan-sharpening is how to correctly select consistent features and propagate them, and properly handle inconsistent ones between PAN and MS modalities. To solve this issue, we propose a Normalization-based Feature Selection and Restitution mechanism, which is capable of filtering out the inconsistent features and promoting to learn the consistent ones. Specifically, we first modulate the PAN feature as the MS style in feature space by AdaIN operation \citeAdaIN. However, such operation inevitably removes the favorable features. We thus propose to distill the effective information from the removed part and restitute it back to the modulated part. To better distillation, we enforce a contrastive learning constraint to close the distance between the restituted feature and the ground truth, and push the removed part away from the ground truth. In this way, the consistent features of PAN images are correctly selected and the inconsistent ones are filtered out, thus relieving the over-transferred artifacts in the process of PAN-guided MS super-resolution. Extensive experiments validate the effectiveness of the proposed network and demonstrate its favorable performance against other state-of-the-art methods. The source code will be released at https://github.com/manman1995/pansharpening.

Supplementary Material

MP4 File (MM22-fp0149.mp4)
Presentation video of Normalization-based Feature Selection and Restitution for Pan-sharpening

References

[1]
Bruno Aiazzi, Stefano Baronti, and Massimo Selva. 2007. Improving component substitution pansharpening through multivariate regression of MS Pan data. IEEE Transactions on Geoscience and Remote Sensing 45, 10 (2007), 3230--3239.
[2]
L. Alparone, L. Wald, J. Chanussot, C. Thomas, P. Gamba, and L. M. Bruce. 2007. Comparison of Pansharpening Algorithms: Outcome of the 2006 GRS-S Data Fusion Contest. IEEE Transactions on Geoscience and Remote Sensing 45, 10 (2007), 3012--3021.
[3]
Coloma Ballester, Vicent Caselles, Laura Igual, Joan Verdera, and Bernard Rougé. 2006. A variational model for P XS image fusion. International Journal of Computer Vision 69, 1 (2006), 43--58.
[4]
Jiajun Cai and Bo Huang. 2021. Super-Resolution-Guided Progressive Pansharpening Based on a Deep Convolutional Neural Network. IEEE Transactions on Geoscience and Remote Sensing 59, 6 (2021), 5206--5220.
[5]
Xiangyong Cao, Yang Chen, Qian Zhao, Deyu Meng, Yao Wang, Dong Wang, and Zongben Xu. 2015. Low-Rank Matrix Factorization under General Mixture Noise Distributions. In 2015 IEEE International Conference on Computer Vision (ICCV). 1493--1501.
[6]
Xiangyong Cao, Xueyang Fu, Danfeng Hong, Zongben Xu, and Deyu Meng. 2021. PanCSC-Net: A Model-Driven Deep Unfolding Method for Pansharpening. IEEE Transactions on Geoscience and Remote Sensing (2021), 1--13.
[7]
Xiangyong Cao, Lin Xu, Deyu Meng, Qian Zhao, and Zongben Xu. 2017. Integration of 3-dimensional discrete wavelet transform and Markov random field for hyperspectral image classification. Neurocomputing 226 (2017), 90--100.
[8]
Xiangyong Cao, Zongben Xu, and Deyu Meng. 2019. Spectral-Spatial Hyperspectral Image Classification via Robust Low-Rank Feature Extraction and Markov Random Field. Remote. Sens. 11, 13 (2019), 1565.
[9]
Xiangyong Cao, Jing Yao, Zongben Xu, and Deyu Meng. 2020. Hyperspectral Image Classification With Convolutional Neural Network and Active Learning. IEEE Transactions on Geoscience and Remote Sensing 58, 7 (2020), 4604--4616.
[10]
Xiangyong Cao, Feng Zhou, Lin Xu, Deyu Meng, Zongben Xu, and John Paisley. 2018. Hyperspectral Image Classification With Markov Random Fields and a Convolutional Neural Network. IEEE Transactions on Image Processing 27, 5 (2018), 2354--2367.
[11]
Wjoseph Carper, Thomasm Lillesand, and Ralphw Kiefer. 1990. The use of intensity-hue-saturation transformations for merging SPOT panchromatic and multispectral image data. Photogrammetric Engineering and remote sensing 56, 4 (1990), 459--467.
[12]
Chen Chen, Yeqing Li, Wei Liu, and Junzhou Huang. 2015. SIRF: Simultaneous Satellite Image Registration and Fusion in a Unified Framework. IEEE Transactions on Image Processing 24, 11 (2015), 4213--4224.
[13]
Xueyang Fu, Zihuang Lin, Yue Huang, and Xinghao Ding. 2019. A variational pan-sharpening with local gradient constraints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10265--10274.
[14]
Ying Fu, Zhiyuan Liang, and Shaodi You. 2021. Bidirectional 3D Quasi-Recurrent Neural Network for Hyperspectral Image Super-Resolution. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 14 (2021), 2674-- 2688.
[15]
Morteza Ghahremani and Hassan Ghassemian. 2016. Nonlinear IHS: A promising method for pan-sharpening. IEEE Geoscience and Remote Sensing Letters 13, 11 (2016), 1606--1610.
[16]
Alan R Gillespie, Anne B Kahle, and Richard E Walker. 1987. Color enhancement of highly correlated images. II. Channel ratio and "chromaticity" transformation techniques. Remote Sensing of Environment 22, 3 (1987), 343--365.
[17]
A. R. Gillespie, A. B. Kahle, and R. E. Walker. 1987. Color enhancement of highly correlated images. II. Channel ratio and "chromaticity" transformation techniques - ScienceDirect. Remote Sensing of Environment 22, 3 (1987), 343--365.
[18]
Juan Mario Haut, Mercedes E. Paoletti, Javier Plaza, Jun Li, and Antonio Plaza. 2018. Active Learning With Convolutional Neural Networks for Hyperspectral Image Classification Using a New Bayesian Approach. IEEE Transactions on Geoscience and Remote Sensing 56, 11 (2018), 6440--6461.
[19]
R. Haydn, G.W. Dalke, J. Henkel, and J. E. Bare. 1982. Application of the IHS color transform to the processing of multisensor data and image enhancement. National Academy of Sciences of the United States of America 79, 13 (1982), 571--577.
[20]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition. 770--778.
[21]
Xun Huang and Serge Belongie. 2017. Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization. In 2017 IEEE International Conference on Computer Vision (ICCV). 1510--1519. https://doi.org/10.1109/ICCV.2017.167
[22]
A. F. Goetz J. R. H. Yuhas and J. M. Boardman. 1992. Discrimination among semiarid landscape endmembers using the spectral angle mapper (SAM) algorithm. Proc. Summaries Annu. JPL Airborne Geosci. Workshop (1992), 147--149.
[23]
Junjun Jiang, Jiayi Ma, Chen Chen, Zhongyuan Wang, Zhihua Cai, and Lizhe Wang. 2018. SuperPCA: A Superpixelwise PCA Approach for Unsupervised Feature Extraction of Hyperspectral Imagery. IEEE Transactions on Geoscience and Remote Sensing 56, 8 (2018), 4581--4593.
[24]
Junjun Jiang, Jiayi Ma, and Xianming Liu. 2020. Multilayer Spectral-Spatial Graphs for Label Noisy Robust Hyperspectral Image Classification. IEEE Transactions on Neural Networks and Learning Systems (2020), 1--14.
[25]
Junjun Jiang, Jiayi Ma, ZhengWang, Chen Chen, and Xianming Liu. 2019. Hyperspectral Image Classification in the Presence of Noisy Labels. IEEE Transactions on Geoscience and Remote Sensing 57, 2 (2019), 851--865.
[26]
Junjun Jiang, He Sun, Xianming Liu, and Jiayi Ma. 2020. Learning Spatial-Spectral Prior for Super-Resolution of Hyperspectral Imagery. IEEE Transactions on Computational Imaging 6 (2020), 1082--1096.
[27]
Kui Jiang, Zhongyuan Wang, Peng Yi, and Junjun Jiang. 2018. A Progressively Enhanced Network for Video Satellite Imagery Superresolution. IEEE Signal Processing Letters 25, 11 (2018), 1630--1634.
[28]
Kui Jiang, Zhongyuan Wang, Peng Yi, Junjun Jiang, Guangcheng Wang, Zhen Han, and Tao Lu. 2019. GAN-Based Multi-level Mapping Network for Satellite Imagery Super-Resolution. In 2019 IEEE International Conference on Multimedia and Expo (ICME). 526--531.
[29]
Kui Jiang, Zhongyuan Wang, Peng Yi, Junjun Jiang, Emily Xiao, and Yuan Yao. 2018. Deep Distillation Recursive Network for Remote Sensing Imagery Super- Resolution. Remote Sensing 10 (10 2018), 1700.
[30]
Kui Jiang, Zhongyuan Wang, Peng Yi, Guangcheng Wang, Tao Lu, and Junjun Jiang. 2019. Edge-Enhanced GAN for Remote Sensing Image Superresolution. IEEE Transactions on Geoscience and Remote Sensing 57, 8 (2019), 5799--5812.
[31]
Muhammad Murtaza Khan, Jocelyn Chanussot, Laurent Condat, and Annick Montanvert. 2008. Indusion: Fusion of multispectral and panchromatic images using the induction scaling technique. IEEE Geoscience and Remote Sensing Letters 5, 1 (2008), 98--102.
[32]
P Kwarteng and A Chavez. 1989. Extracting spectral contrast in Landsat Thematic Mapper image data using selective principal component analysis. Photogrammetric Engineering and remote sensing 55, 339--348 (1989), 1.
[33]
C.A. Laben and B.V. Brower. 2000. Process for Enhancing the Spatial Resolution of Multispectral Imagery Using Pan-Sharpening. US Patent 6011875A (2000).
[34]
Craig A Laben and Bernard V Brower. 2000. Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. US Patent 6,011,875.
[35]
W. Liao, H. Xin, F. V. Coillie, G. Thoonen, and W. Philips. 2017. Two-stage fusion of thermal hyperspectral and visible RGB image by PCA and guided filter. In Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing.
[36]
J. G. Liu. 2000. Smoothing filter-based intensity modulation: A spectral preserve image fusion technique for improving spatial details. International Journal of Remote Sensing 21, 18 (2000), 3461--3472.
[37]
Jiayi Ma, Han Xu, Junjun Jiang, Xiaoguang Mei, and Xiao-Ping Zhang. 2020. DDcGAN: A Dual-Discriminator Conditional Generative Adversarial Network for Multi-Resolution Image Fusion. IEEE Transactions on Image Processing 29 (2020), 4980--4995.
[38]
Jiayi Ma, Wei Yu, Chen Chen, Pengwei Liang, Xiaojie Guo, and Junjun Jiang. 2020. Pan-GAN: An unsupervised pan-sharpening method for remote sensing image fusion. Information Fusion 62 (2020), 110--120.
[39]
SG Mallat. 1989. ATheory for Multiresolution Signal Decomposition: TheWavelet Representation. IEEE Transactions on Pattern Analysis and Machine Intelligence 11, 7 (1989), 674--693.
[40]
Giuseppe Masi, Davide Cozzolino, Luisa Verdoliva, and Giuseppe Scarpa. 2016. Pansharpening by Convolutional Neural Networks. Remote Sensing 8, 7 (2016).
[41]
Jorge Nunez, Xavier Otazu, Octavi Fors, Albert Prades, Vicenc Pala, and Roman Arbiol. 1999. Multiresolution-based image fusion with additive wavelet decomposition. IEEE Transactions on Geoscience and Remote sensing 37, 3 (1999), 1204--1211.
[42]
Robert A Schowengerdt. 1980. Reconstruction of multispatial, multispectral image data using spatial frequency content. Photogrammetric Engineering and Remote Sensing 46, 10 (1980), 1325--1334.
[43]
Vijay P. Shah, Nicolas H. Younan, and Roger L. King. 2008. An Efficient Pan- Sharpening Method via a Combined Adaptive PCA Approach and Contourlets. IEEE Transactions on Geoscience and Remote Sensing 46, 5 (2008), 1323--1335.
[44]
Xin Tian, Yuerong Chen, Changcai Yang, Xun Gao, and Jiayi Ma. 2020. A Variational Pansharpening Method Based on Gradient Sparse Representation. IEEE Signal Processing Letters 27 (2020), 1180--1184.
[45]
Xin Tian, Yuerong Chen, Changcai Yang, and Jiayi Ma. 2021. Variational Pansharpening by Exploiting Cartoon-Texture Similarities. IEEE Transactions on Geoscience and Remote Sensing (2021), 1--16.
[46]
Xin Tian, Kun Li, ZhongyuanWang, and Jiayi Ma. 2021. VP-Net: An Interpretable Deep Network for Variational Pansharpening. IEEE Transactions on Geoscience and Remote Sensing (2021), 1--16.
[47]
Gemine Vivone, Luciano Alparone, Jocelyn Chanussot, Mauro Dalla Mura, Andrea Garzelli, Giorgio A Licciardi, Rocco Restaino, and Lucien Wald. 2014. A critical comparison among pansharpening algorithms. IEEE Transactions on Geoscience and Remote Sensing 53, 5 (2014), 2565--2586.
[48]
Lucien Wald, Thierry Ranchin, and Marc Mangolini. 1997. Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images. Photogrammetric Engineering and Remote Sensing 63 (11 1997), 691--699.
[49]
Xinya Wang, Jiayi Ma, and Junjun Jiang. 2021. Hyperspectral Image Super- Resolution via Recurrent Feedback Embedding and Spatial-Spectral Consistency Regularization. IEEE Transactions on Geoscience and Remote Sensing (2021), 1--13.
[50]
XiaoWu, Ting-Zhu Huang, Liang-Jian Deng, and Tian-Jing Zhang. 2021. Dynamic Cross Feature Fusion for Remote Sensing Pansharpening. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 14687--14696.
[51]
Zhong-Cheng Wu, Ting-Zhu Huang, Liang-Jian Deng, Jin-Fan Hu, and Gemine Vivone. 2021. VONet: An Adaptive Approach Using Variational Optimization and Deep Learning for Panchromatic Sharpening. IEEE Transactions on Geoscience and Remote Sensing (2021), 1--16.
[52]
Han Xu, Jiayi Ma, Zhenfeng Shao, Hao Zhang, Junjun Jiang, and Xiaojie Guo. 2021. SDPNet: A Deep Network for Pan-Sharpening With Enhanced Information Representation. IEEE Transactions on Geoscience and Remote Sensing 59, 5 (2021), 4120--4134.
[53]
Shuang Xu, Jiangshe Zhang, Zixiang Zhao, Kai Sun, Junmin Liu, and Chunxia Zhang. 2021. Deep Gradient Projection Networks for Pan-sharpening. In IEEE Conference on Computer Vision and Pattern Recognition. 1366--1375.
[54]
Keyu Yan, Man Zhou, Liu Liu, Chengjun Xie, and Danfeng Hong. 2022. When Pansharpening Meets Graph Convolution Network and Knowledge Distillation. IEEE Transactions on Geoscience and Remote Sensing 60 (2022), 1--15. https://doi.org/10.1109/TGRS.2022.3168192
[55]
Gang Yang, Man Zhou, Keyu Yan, Aiping Liu, Xueyang Fu, and Fan Wang. 2022. Memory-Augmented Deep Conditional Unfolding Network for Pan-Sharpening. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 1788--1797.
[56]
Junfeng Yang, Xueyang Fu, Yuwen Hu, Yue Huang, Xinghao Ding, and John Paisley. 2017. PanNet: A deep network architecture for pan-sharpening. In IEEE International Conference on Computer Vision. 5449--5457.
[57]
Qiangqiang Yuan, Yancong Wei, Xiangchao Meng, Huanfeng Shen, and Liangpei Zhang. 2018. A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 11, 3 (2018), 978--989.
[58]
Q. Yuan, Y. Wei, X. Meng, H. Shen, and L. Zhang. 2018. A Multiscale and Multidepth Convolutional Neural Network for Remote Sensing Imagery Pan- Sharpening. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 11, 3 (2018), 978--989.
[59]
Hao Zhang and Jiayi Ma. 2021. GTP-PNet: A residual learning network based on gradient transformation prior for pansharpening. ISPRS Journal of Photogrammetry and Remote Sensing 172 (2021), 223--239.
[60]
Man Zhou, Xueyang Fu, Jie Huang, Feng Zhao, Aiping Liu, and Rujing Wang. 2022. Effective Pan-Sharpening With Transformer and Invertible Neural Network. IEEE Transactions on Geoscience and Remote Sensing 60 (2022), 1--15. https://doi.org/10.1109/TGRS.2021.3137967
[61]
Man Zhou, Jie Huang, Yanchi Fang, Xueyang Fu, and Aiping Liu. 2022. Pan-Sharpening with Customized Transformer and Invertible Neural Network. AAAI Press.
[62]
Man Zhou, Zeyu Xiao, Xueyang Fu, Aiping Liu, Gang Yang, and Zhiwei Xiong. 2021. Unfolding Taylor's Approximations for Image Restoration. In NeurIPS.
[63]
Man Zhou, Keyu Yan, Jie Huang, Zihe Yang, Xueyang Fu, and Feng Zhao. 2022. Mutual Information-Driven Pan-Sharpening. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 1798--1808.
[64]
Man Zhou, Keyu Yan, Jinshan Pan, Wenqi Ren, Qiaokang Xie, and Xiangyong Cao. 2022. Memory-augmented Deep Unfolding Network for Guided Image Super-resolution. ArXiv abs/2203.04960 (2022).
[65]
Tian-Jing Zhang Xiaoxu Jin Zi-Rong Jin, Liang-Jian Deng. 2021. BAM: Bilateral Activation Mechanism for Image Fusion. Proceedings of the 29th ACMInternational Conference on Multimedia (ACM MM) (2021).

Cited By

View all
  • (2025)DMPNet: dual-path and multi-scale pansharpening networkFrontiers in Computer Science10.3389/fcomp.2024.14559636Online publication date: 17-Jan-2025
  • (2024)Multi-Frequency Spectral–Spatial Interactive Enhancement Fusion Network for Pan-SharpeningElectronics10.3390/electronics1314280213:14(2802)Online publication date: 16-Jul-2024
  • (2024)VSDM: Variable-Scale Diffusion Model Based on Dynamic Condition Guidance for PansharpeningIEEE Transactions on Geoscience and Remote Sensing10.1109/TGRS.2024.350485762(1-12)Online publication date: 2024
  • Show More Cited By

Index Terms

  1. Normalization-based Feature Selection and Restitution for Pan-sharpening

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MM '22: Proceedings of the 30th ACM International Conference on Multimedia
    October 2022
    7537 pages
    ISBN:9781450392037
    DOI:10.1145/3503161
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 10 October 2022

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. contrastive learning
    2. normalization
    3. pan-sharpening

    Qualifiers

    • Research-article

    Funding Sources

    • Anhui Provincial Natural Science Foundation

    Conference

    MM '22
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)39
    • Downloads (Last 6 weeks)2
    Reflects downloads up to 16 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)DMPNet: dual-path and multi-scale pansharpening networkFrontiers in Computer Science10.3389/fcomp.2024.14559636Online publication date: 17-Jan-2025
    • (2024)Multi-Frequency Spectral–Spatial Interactive Enhancement Fusion Network for Pan-SharpeningElectronics10.3390/electronics1314280213:14(2802)Online publication date: 16-Jul-2024
    • (2024)VSDM: Variable-Scale Diffusion Model Based on Dynamic Condition Guidance for PansharpeningIEEE Transactions on Geoscience and Remote Sensing10.1109/TGRS.2024.350485762(1-12)Online publication date: 2024
    • (2024)ConvGRU-Based Multiscale Frequency Fusion Network for PAN-MS Joint ClassificationIEEE Transactions on Geoscience and Remote Sensing10.1109/TGRS.2024.341537162(1-15)Online publication date: 2024
    • (2024)Pan-Sharpening With Wavelet-Enhanced High-Frequency InformationIEEE Transactions on Geoscience and Remote Sensing10.1109/TGRS.2024.336716562(1-14)Online publication date: 2024
    • (2024)Progressive Reconstruction Network With Adaptive Frequency Adjustment for PansharpeningIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing10.1109/JSTARS.2024.345231117(17382-17397)Online publication date: 2024
    • (2023)CTCP: Cross Transformer and CNN for PansharpeningProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3613815(3003-3011)Online publication date: 26-Oct-2023
    • (2023)Multi-scale Spatial-Spectral Attention Guided Fusion Network for PansharpeningProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3613814(3346-3354)Online publication date: 26-Oct-2023
    • (2023)U2Net: A General Framework with Spatial-Spectral-Integrated Double U-Net for Image FusionProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612084(3219-3227)Online publication date: 26-Oct-2023
    • (2023)Uncertainty-Driven Dynamic Degradation Perceiving and Background Modeling for Efficient Single Image DesnowingProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612003(4269-4280)Online publication date: 26-Oct-2023
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media