Zhenglin Tang
ABSTRACT
The visible and near-infrared image fusion aims to generate an image that integrates complementary information from images captured in different spectral bands. However, existing fusion methods either only focus on the fusion of spatial information, or fuse spatial and spectral information without decoupling, resulting in undesirable effects such as halo artifacts, information loss, and inferior visual quality. To address these issues, we propose a Spatial-Spectral Decoupling Fusion (SSDF) framework that can effectively fuse the spatial and spectral information of visible and near-infrared images. The SSDF framework decomposes the image pairs into two main branches: the Spatial Feature Enhancement (SFE) branch and the Spectral Characteristic Preservation (SCP) branch. The SFE branch enhances the salient details in the fused image by exploiting the contrast between spatial features and generating region-based fusion weights, while the SCP branch preserves the intrinsic spectral characteristics of the scene by fusing the reflectance characteristics of visible and near-infrared images. The final image is obtained by combining the spatial and spectral information. We conduct extensive experiments to show that our SSDF method can achieve superior fusion performance in subjective visual quality and objective metrics compared with state-of-the-art methods.
- Mohamed Awad, Ahmed Elliethy, and Hussein A Aly. 2019. Adaptive near-infrared and visible fusion for fast image enhancement. IEEE Transactions on Computational Imaging 6 (2019), 408–418.Google Scholar
Cross Ref
- C Bellotti, F Bellotti, A De Gloria, L Andreone, and Michele Mariani. 2004. Developing a near infrared based night vision system. In IEEE Intelligent Vehicles Symposium, 2004. IEEE, 686–691.Google ScholarCross Ref
- Edward R Dougherty and Roberto A Lotufo. 2003. Hands-on morphological image processing. Vol. 59. SPIE press.Google Scholar
- Chen Feng, Shaojie Zhuo, Xiaopeng Zhang, Liang Shen, and Sabine Süsstrunk. 2013. Near-infrared guided color image dehazing. In 2013 IEEE international conference on image processing. IEEE, 2363–2367.Google ScholarCross Ref
- Bo-Cai Gao. 1996. NDWI—A normalized difference water index for remote sensing of vegetation liquid water from space. Remote sensing of environment 58, 3 (1996), 257–266.Google Scholar
- Nadine Gobron, Bernard Pinty, and Michel M Verstraete. 1997. Theoretical limits to the estimation of the leaf area index on the basis of visible and near-infrared remote sensing data. IEEE Transactions on Geoscience and Remote Sensing 35, 6 (1997), 1438–1445.Google ScholarCross Ref
- Yu Han, Yunze Cai, Yin Cao, and Xiaoming Xu. 2013. A new image fusion performance metric based on visual information fidelity. Information fusion 14, 2 (2013), 127–135.Google Scholar
- David Hasler and Sabine E Suesstrunk. 2003. Measuring colorfulness in natural images. In Human vision and electronic imaging VIII, Vol. 5007. SPIE, 87–95.Google Scholar
- Kaiming He, Jian Sun, and Xiaoou Tang. 2010. Single image haze removal using dark channel prior. IEEE transactions on pattern analysis and machine intelligence 33, 12 (2010), 2341–2353.Google Scholar
- María Herrera-Arellano, Hayde Peregrina-Barreto, and Iván Terol-Villalobos. 2021. Visible-NIR image fusion based on top-hat transform. IEEE Transactions on Image Processing 30 (2021), 4962–4972.Google ScholarCross Ref
- María A Herrera-Arellano, Hayde Peregrina-Barreto, and Iván Terol-Villalobos. 2019. Color outdoor image enhancement by V-NIR fusion and weighted luminance. In 2019 IEEE International Autumn Meeting on Power, Electronics and Computing (ROPEC). IEEE, 1–6.Google ScholarCross Ref
- Chan-Gi Im, Dong-Min Son, Hyuk-Ju Kwon, and Sung-Hak Lee. 2022. Tone Image Classification and Weighted Learning for Visible and NIR Image Fusion. Entropy 24, 10 (2022), 1435.Google ScholarCross Ref
- Dong-Won Jang and Rae-Hong Park. 2017. Colour image dehazing using near-infrared fusion. IET Image Processing 11, 8 (2017), 587–594.Google ScholarCross Ref
- Jionghui Jiang, Fen Liu, Yingying Xu, Hui Huang, 2019. Multi-spectral RGB-NIR image classification using double-channel CNN. IEEE Access 7 (2019), 20607–20613.Google ScholarCross Ref
- Hyungjoo Jung, Youngjung Kim, Hyunsung Jang, Namkoo Ha, and Kwanghoon Sohn. 2020. Unsupervised deep image fusion with structure tensor representations. IEEE Transactions on Image Processing 29 (2020), 3845–3858.Google ScholarDigital Library
- FJ Kriegler, WA Malila, RF Nalepka, and W Richardson. 1969. Preprocessing transformations and their effects on multispectral recognition. Remote sensing of environment, VI (1969), 97.Google Scholar
- Rahul Kumar, Brajesh Kumar Kaushik, and R Balasubramanian. 2019. Multispectral transmission map fusion method and architecture for image dehazing. IEEE Transactions on Very Large Scale Integration (VLSI) Systems 27, 11 (2019), 2693–2697.Google ScholarDigital Library
- Hyuk-Ju Kwon and Sung-Hak Lee. 2021. Visible and near-infrared image acquisition and fusion for night surveillance. Chemosensors 9, 4 (2021), 75.Google ScholarCross Ref
- Shutao Li, Xudong Kang, Leyuan Fang, Jianwen Hu, and Haitao Yin. 2017. Pixel-level image fusion: A survey of the state of the art. information Fusion 33 (2017), 100–112.Google Scholar
- Zhuo Li, Hai-Miao Hu, Wei Zhang, Shiliang Pu, and Bo Li. 2020. Spectrum characteristics preserved visible and near-infrared image fusion algorithm. IEEE Transactions on Multimedia 23 (2020), 306–319.Google ScholarCross Ref
- Yu Liu, Xun Chen, Juan Cheng, and Hu Peng. 2017. A medical image fusion method based on convolutional neural networks. In 2017 20th international conference on information fusion (Fusion). IEEE, 1–7.Google ScholarCross Ref
- Yu Liu, Lei Wang, Juan Cheng, Chang Li, and Xun Chen. 2020. Multi-focus image fusion: A survey of the state of the art. Information Fusion 64 (2020), 71–91.Google ScholarCross Ref
- Yue M Lu, Clément Fredembach, Martin Vetterli, and Sabine Süsstrunk. 2009. Designing color filter arrays for the joint capture of visible and near-infrared images. In 2009 16th IEEE International Conference on Image Processing (ICIP). IEEE, 3797–3800.Google ScholarCross Ref
- Yun Luo, Jeffrey Remillard, and Dieter Hoetzer. 2010. Pedestrian detection in near-infrared night vision system. In 2010 IEEE Intelligent Vehicles Symposium. IEEE, 51–58.Google ScholarCross Ref
- Aziz Makandar and Bhagirathi Halalli. 2015. Image enhancement techniques using highpass and lowpass filters. International Journal of Computer Applications 109, 14 (2015), 12–15.Google ScholarCross Ref
- Harvey B Mitchell. 2010. Image fusion: theories, techniques and applications. Springer Science & Business Media.Google Scholar
- Nirmala Paramanandham and Kishore Rajendiran. 2018. Multi sensor image fusion for surveillance applications using hybrid image fusion algorithm. Multimedia Tools and Applications 77 (2018), 12405–12436.Google ScholarDigital Library
- Akikazu Sakudo. 2016. Near-infrared spectroscopy for medical applications: Current status and future perspectives. Clinica Chimica Acta 455 (2016), 181–188.Google ScholarCross Ref
- Neda Salamati, Zahra Sadeghipoor, and Sabine Süsstrunk. 2011. Analyzing near-infrared images for utility assessment. In Human Vision and Electronic Imaging XVI, Vol. 7865. SPIE, 62–70.Google Scholar
- James R Shell II 2005. Polarimetric remote sensing in the visible to near infrared. (2005).Google Scholar
- Chang-Hwan Son and Xiao-Ping Zhang. 2016. Near-infrared image dehazing via color regularization. arXiv preprint arXiv:1610.00175 (2016).Google Scholar
- Chang-Hwan Son and Xiao-Ping Zhang. 2017. Near-infrared coloring via a contrast-preserving mapping model. IEEE Transactions on Image Processing 26, 11 (2017), 5381–5394.Google ScholarDigital Library
- Tania Stathaki. 2011. Image fusion: algorithms and applications. Elsevier.Google Scholar
- Ashish V Vanmali and Vikram M Gadre. 2017. Visible and NIR image fusion using weight-map-guided Laplacian–Gaussian pyramid for improving scene visibility. Sādhanā 42 (2017), 1063–1082.Google ScholarCross Ref
- Ashish V Vanmali, Samrudha G Kelkar, and Vikram M Gadre. 2015. A novel approach for image dehazing combining visible-NIR images. In 2015 Fifth National Conference on Computer Vision, Pattern Recognition, Image Processing and Graphics (NCVPRIPG). IEEE, 1–4.Google ScholarCross Ref
- Farshad G Veshki and Sergiy A Vorobyov. 2022. Convolutional Simultaneous Sparse Approximation with Applications to RGB-NIR Image Fusion. In 2022 56th Asilomar Conference on Signals, Systems, and Computers. IEEE, 872–876.Google ScholarCross Ref
- Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600–612.Google ScholarDigital Library
- Pu Xia, Xiaolai Chen, and Xiaohui Gao. 2021. Polarimetric dehazing utilizing Vis-NIR spectral image fusion. In 2021 International Conference on Intelligent Computing, Automation and Systems (ICICAS). IEEE, 48–54.Google ScholarCross Ref
- Li Xu, Qiong Yan, Yang Xia, and Jiaya Jia. 2012. Structure extraction from texture via relative total variation. ACM transactions on graphics (TOG) 31, 6 (2012), 1–10.Google Scholar
- Costas S Xydeas, Vladimir Petrovic, 2000. Objective image fusion performance measure. Electronics letters 36, 4 (2000), 308–309.Google Scholar
- Qiong Yan, Xiaoyong Shen, Li Xu, Shaojie Zhuo, Xiaopeng Zhang, Liang Shen, and Jiaya Jia. 2013. Cross-field joint image restoration via scale map. In Proceedings of the IEEE International Conference on Computer Vision. 1537–1544.Google ScholarDigital Library
- Guanyu Zhang, Beichen Sun, Yuehan Qi, and Yang Liu. 2022. Visible and Near Infrared Image Fusion Based on Texture Information. arXiv preprint arXiv:2207.10953 (2022).Google Scholar
- Xingchen Zhang. 2021. Deep learning-based multi-focus image fusion: A survey and a comparative study. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 9 (2021), 4819–4838.Google Scholar
- Yufeng Zheng. 2011. Image fusion and its applications. BoD–Books on Demand.Google Scholar
Index Terms
- A Spatial-Spectral Decoupling Fusion Framework for Visible and Near-Infrared Images
Recommendations
Fusion method for infrared and visible images based on improved quantum theory model
Quantum theory model (QTM) owns a superior characteristic to turn a complex problem into the form of the linear combination of several much simpler components. Therefore, a novel fusion method based on improved quantum theory model (IQTM) is proposed in ...
A Fourier Transform-based Approach to Fusion High Spatial Resolution Remote Sensing Images
ICVGIP '08: Proceedings of the 2008 Sixth Indian Conference on Computer Vision, Graphics & Image ProcessingThe modern remote sensing imaging sensors, like those in the IKONOS and QuickBird satellites, are capable of generating panchromatic images with one meter spatial resolution and multiespectral images with good spectral information. The principal ...
Fusion of near-infrared and visible images based on saliency-map-guided multi-scale transformation decomposition
AbstractIn this research, we propose a near-infrared (NIR) and visible (VIS) image fusion method based on saliency-map-guided multi-scale transform decomposition (SMG-MST) to solve the problem of color distortion. Although the existing NIR and VIS image ...
Comments