Skip to main content
SpringerLink
Log in

Superpixel-based adaptive salient region analysis for infrared and visible image fusion

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Infrared and visible image fusion aims to highlight the infrared target and preserve valuable texture details as much as possible. However, the infrared target needs to be more apparent in most image fusion methods. A large amount of infrared noise remains in the fusion results, significantly reducing the proportion of valuable texture details in the fusion results. How to highlight the salient of infrared targets, lower noise, and retain more valuable texture details in the fusion results still need to be solved. We propose an adaptive salient region analysis method based on superpixels (SSRA) for infrared and visible fusion to solve this problem. This method uses salient region analysis based on superpixels to highlight the salience region effectively. We design a texture detail fusion method based on brightness analysis of the visible image to suppress noise and keep more meaningful texture detail information. The experimental results show that our proposed method performs better in subjective vision and quantitative evaluation than some advanced methods. In addition, we also demonstrate that SSRA is capable of supporting high-level visual tasks well. Our code is publicly available at: https://github.com/VCMHE/SSRA.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig.10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. He K, Gong J, Xie L et al (2021) Regions preserving edge enhancement for multisensor-based medical image fusion. IEEE Trans Instrum Meas 70:1–13

    Google Scholar 

  2. He K, Zhang X, Xu D, et al (2022) Fidelity-driven optimization reconstruction and details preserving guided fusion for multi-modality medical image. IEEE Trans Multimedia

  3. Yin W, He K, Xu D et al (2022) Adaptive enhanced infrared and visible image fusion using hybrid decomposition and coupled dictionary. Neural Comput Appl 34:20831–20849

    Article  Google Scholar 

  4. Yin W, He K, Xu D et al (2022) Significant target analysis and detail preserving based infrared and visible image fusion. Infrared Phys Technol 121:104041

    Article  Google Scholar 

  5. Jiang Q, Lee S, Zeng X et al (2022) A multifocus image fusion scheme based on similarity measure of transformed isosceles triangles between intuitionistic fuzzy sets. IEEE Trans Instrum Meas 71:1–15

    Google Scholar 

  6. Jin X, Jiang Q, Yao S et al (2017) A survey of infrared and visual image fusion methods. Infrared Phys Technol 85:478–501

    Article  Google Scholar 

  7. Noori FM, Riegler M, Uddin MZ, Torresen J (2020) Human activity recognition from multiple sensors data using multi-fusion representations and CNNs. ACM Trans Multimedia Comput Commun Appl (TOMM) 16:1–19

  8. Zhu Z, He X, Qi G et al (2023) Brain tumor segmentation based on the fusion of deep semantics and edge information in multimodal MRI. Inf Fusion 91:376–387. https://doi.org/10.1016/j.inffus.2022.10.022

    Article  Google Scholar 

  9. Yin M, Duan P, Liu W, Liang X (2017) A novel infrared and visible image fusion algorithm based on shift-invariant dual-tree complex shearlet transform and sparse representation. Neurocomputing 226:182–191

    Article  Google Scholar 

  10. Yang Z, Chen Y, Le Z, Ma Y (2021) GANFuse: a novel multi-exposure image fusion method based on generative adversarial networks. Neural Comput Appl 33:6133–6145

    Article  Google Scholar 

  11. Singh S, Mittal N, Singh H (2022) A feature level image fusion for IR and visible image using mNMRA based segmentation. Neural Comput Appl 34:8137–8154

    Article  Google Scholar 

  12. Nandhakumar N, Aggarwal JK (1988) Integrated analysis of thermal and visual images for scene interpretation. IEEE Trans Pattern Anal Mach Intell 10:469–481

    Article  Google Scholar 

  13. Morris NJ, Avidan S, Matusik W, Pfister H (2007) Statistics of infrared images. In: 2007 IEEE conference on computer vision and pattern recognition. IEEE, New York, pp 1–7

  14. Li L, Li H, Dang E, Liu B (2013) Compressive sensing method for recognizing cat-eye effect targets. Appl Opt 52:7033–7039

    Article  Google Scholar 

  15. Li L, Li H, Li T, Gao F (2014) Infrared small target detection in compressive domain. Electron Lett 50:510–512

    Article  Google Scholar 

  16. Kansal K, Subramanyam AV, Wang Z, Satoh S (2020) SDL: spectrum-disentangled representation learning for visible-infrared person re-identification. IEEE Trans Circuits Syst Video Technol 30:3422–3432

    Article  Google Scholar 

  17. Ma J, Zhou Z, Wang B, Zong H (2017) Infrared and visible image fusion based on visual saliency map and weighted least square optimization. Infrared Phys Technol 82:8–17

    Article  Google Scholar 

  18. Ma J, Chen C, Li C, Huang J (2016) Infrared and visible image fusion via gradient transfer and total variation minimization. Information Fusion 31:100–109

    Article  Google Scholar 

  19. Bavirisetti DP, Dhuli R (2015) Fusion of infrared and visible sensor images based on anisotropic diffusion and Karhunen-Loeve transform. IEEE Sens J 16:203–209

    Article  Google Scholar 

  20. Bavirisetti DP, Dhuli R (2016) Two-scale image fusion of visible and infrared images using saliency detection. Infrared Phys Technol 76:52–64

    Article  Google Scholar 

  21. Chen J, Li X, Luo L et al (2020) Infrared and visible image fusion based on target-enhanced multiscale transform decomposition. Inf Sci 508:64–78

    Article  Google Scholar 

  22. Burt PJ, Adelson EH (1987) The Laplacian pyramid as a compact image code. In: Readings in computer vision. Elsevier, Amsredam, pp 671–679

  23. Niu Y, Xu S, Wu L, Hu W (2012) Airborne infrared and visible image fusion for target perception based on target region segmentation and discrete wavelet transform. Math Problems Eng 2012:1–10

    Google Scholar 

  24. Fletcher P, Sangwine SJ (2017) The development of the quaternion wavelet transform. Signal Process 136:2–15

    Article  Google Scholar 

  25. Da Cunha AL, Zhou J, Do MN (2006) The nonsubsampled contourlet transform: theory, design, and applications. IEEE Trans Image Process 15:3089–3101

    Article  Google Scholar 

  26. Tan W, Zhou H, Song J et al (2019) Infrared and visible image perceptive fusion through multi-level Gaussian curvature filtering image decomposition. Appl Opt 58:3064–3073

    Article  Google Scholar 

  27. Li H, Qi X, Xie W (2020) Fast infrared and visible image fusion with structural decomposition. Knowledge-Based Syst 204:106182. https://doi.org/10.1016/j.knosys.2020.106182

  28. Wright J, Yang AY, Ganesh A et al (2008) Robust face recognition via sparse representation. IEEE Trans Pattern Anal Mach Intell 31:210–227

    Article  Google Scholar 

  29. Liu G, Lin Z, Yan S et al (2012) Robust recovery of subspace structures by low-rank representation. IEEE Trans Pattern Anal Mach Intell 35:171–184

    Article  Google Scholar 

  30. Li H, Wu X-J, Kittler J (2020) MDLatLRR: a novel decomposition method for infrared and visible image fusion. IEEE Trans Image Process 29:4733–4746

    Article  MATH  Google Scholar 

  31. Wang B, Zou Y, Zhang L et al (2022) Multimodal super-resolution reconstruction of infrared and visible images via deep learning. Opt Lasers Eng 156:107078

    Article  Google Scholar 

  32. Liu Y, Chen X, Peng H, Wang Z (2017) Multi-focus image fusion with a deep convolutional neural network. Inform Fusion 36:191–207

    Article  Google Scholar 

  33. Guo X, Nie R, Cao J et al (2019) FuseGAN: learning to fuse multi-focus image via conditional generative adversarial network. IEEE Trans Multimedia 21:1982–1996

    Article  Google Scholar 

  34. Zhang H, Ma J (2021) SDNet: a versatile squeeze-and-decomposition network for real-time image fusion. Int J Comput Vis 129:2761–2785

    Article  Google Scholar 

  35. Xu H, Ma J, Jiang J et al (2022) U2Fusion: a unified unsupervised image fusion network. IEEE Trans Pattern Anal Mach Intell 44:502–518. https://doi.org/10.1109/TPAMI.2020.3012548

    Article  Google Scholar 

  36. Guo Y, Chen J, Wang J et al (2020) Closed-loop matters: Dual regression networks for single image super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5407–5416

  37. Achanta R, Shaji A, Smith K et al (2012) SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans Pattern Anal Mach Intell 34:2274–2282

    Article  Google Scholar 

  38. Lee H, Jeon J, Kim J, Lee S (2017) Structure-texture decomposition of images with interval gradient. In: Computer graphics forum. Wiley Online Library, New York, pp 262–274

  39. Zhai Y, Shah M (2006) Visual attention detection in video sequences using spatiotemporal cues. In: Proceedings of the 14th ACM international conference on Multimedia. pp 815–824

  40. Toet, Alexander (2014) TNO image fusion dataset. figshare. Dataset. https://doi.org/10.6084/m9.figshare.1008029.v2

  41. Li H, Wu X-J (2018) DenseFuse: a fusion approach to infrared and visible images. IEEE Trans Image Process 28:2614–2623

    Article  MathSciNet  Google Scholar 

  42. Zhang Y, Liu Y, Sun P et al (2020) IFCNN: a general image fusion framework based on convolutional neural network. Information Fusion 54:99–118

    Article  Google Scholar 

  43. Ha Q, Watanabe K, Karasawa T et al (2017) MFNet: towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes. In: 2017 IEEE/RSJ international conference on intelligent robots and systems, IROS 2017, Vancouver, BC, Canada, September 24–28, 2017. IEEE, New York, pp 5108–5115

  44. Qu G, Zhang D, Yan P (2002) Information measure for performance of image fusion. Electron Lett 38:1

    Article  Google Scholar 

  45. Sheikh HR, Bovik AC, De Veciana G (2005) An information fidelity criterion for image quality assessment using natural scene statistics. IEEE Trans Image Process 14:2117–2128

    Article  Google Scholar 

  46. Sheikh HR, Bovik AC (2006) Image information and visual quality. IEEE Trans Image Process 15:430–444

    Article  Google Scholar 

  47. Chen Y, Blum RS (2009) A new automated quality assessment algorithm for image fusion. Image Vis Comput 27:1421–1432

    Article  Google Scholar 

  48. Eskicioglu AM, Fisher PS (1995) Image quality measures and their performance. IEEE Trans Commun 43:2959–2965

    Article  Google Scholar 

  49. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 779–788

  50. Qin X, Zhang Z, Huang C et al (2020) U2-Net: going deeper with nested U-structure for salient object detection. Pattern Recogn 106:107404

    Article  Google Scholar 

Download references

Funding

This work was supported in part by the provincial major science and technology special plan projects under Grant 202202AD080003, in part by the National Natural Science Foundation of China under Grant 62202416, Grant 62162068, Grant 62172354, Grant 62162065, in part by the Yunnan Province Ten Thousand Talents Program and Yunling Scholars Special Project under Grant YNWR-YLXZ-2018-022, in part by the Yunnan Provincial Science and Technology Department-Yunnan University “Double First Class” Construction Joint Fund Project under Grant No. 2019FY003012, in part by the Science Research Fund Project of Yunnan Provincial Department of Education under grant 2021Y027, in part by the Graduate Research and Innovation Foundation of Yunnan University ZC-22222977.

Author information

Authors and Affiliations

  1. School of Information Science and Engineering, Yunnan University, Kunming, 650091, China

    Chengzhou Li, Kangjian He, Dan Xu, Dapeng Tao, Hongzhen Shi & Wenxia Yin

  2. Yunnan Union Vision Innovation Technology Co Ltd, Kunming, 650091, China

    Xu Lin

Authors
  1. Chengzhou Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kangjian He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dan Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dapeng Tao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xu Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hongzhen Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Wenxia Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kangjian He.

Ethics declarations

Conflict of interest

The authors declare that there they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, C., He, K., Xu, D. et al. Superpixel-based adaptive salient region analysis for infrared and visible image fusion. Neural Comput & Applic 35, 22511–22529 (2023). https://doi.org/10.1007/s00521-023-08916-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08916-z

Keywords

Search

Navigation