Skip to main content
SpringerLink
Log in

A DCT-based multiscale framework for 2D greyscale image fusion using morphological differential features

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Image fusion refers to the process of synergistic combination of useful sensory information from multiple images to synthesize a composite image with greater information content and increased practical value. It aims to maximize pertinent information specific to a sensor while minimizing uncertainty and redundancy in the fused output. In this paper, the authors have proposed a simple yet cohesive framework for 2D greyscale image fusion using morphological differential features. The features are extracted using morphological open–close filters applied at multiple scales using an isotropic structuring element which brings out categorical bright and dark features from the source images. At each scale, the bright (and dark) differential features are mutually compared using higher-valued AC coefficients obtained in the DCT domain within a block. The scale-specific fused features are recursively added to form an image containing high-frequency information from all conceivable scales. The fused image is achieved by superimposing the cumulative feature image onto a suitable base image. The base image is obtained by using a morphological weighted version of pseudomedian filter over the source images using the largest homothetic of the structuring element. The superiority of the framework is empirically verified in different domains of fusion, i.e. multi-focus, multi-sensor, multi-exposure, and multi-spectral image fusion. The proposed approach has surpassed the state-of-the-art unified fusion algorithms in terms of qualitative and quantitative evaluation with a perfect resource-time trade-off. Furthermore, the proposed method has been extended to greyscale–colour and colour–colour image pairs qualifying it for anatomical–functional image fusion.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data Availability

The datasets generated during and/or analysed during the current study are available in https://sites.google.com/view/durgaprasadbavirisetti/datasets,http://glcf.umd.edu/data/ikonos/ and https://www.med.harvard.edu/aanlib/.

References

  1. Aiazzi, B., Baronti, S., Selva, M.: Improving component substitution pansharpening through multivariate regression of ms \(+\) pan data. IEEE Trans. Geosci. Remote Sens. 45(10), 3230–3239 (2007)

    Google Scholar 

  2. Amin-Naji, M., Aghagolzadeh, A.: Multi-focus image fusion using vol and eol in dct domain. arXiv preprint arXiv:1710.06511 (2017)

  3. Amin-Naji, M., Ranjbar-Noiey, P., Aghagolzadeh, A.: Multi-focus image fusion using singular value decomposition in dct domain. In: 2017 10th Iranian Conference on Machine Vision and Image Processing (MVIP), IEEE, pp. 45–51 (2017)

  4. Aslantas, V., Bendes, E.: A new image quality metric for image fusion: the sum of the correlations of differences. AEU Int. J. Electron. Commun. 69(12), 1890–1896 (2015)

    Google Scholar 

  5. Bai, X.: Morphological image fusion using the extracted image regions and details based on multi-scale top-hat transform and toggle contrast operator. Digit. Signal Process. 23(2), 542–554 (2013)

    MathSciNet  Google Scholar 

  6. Bai, X.: Infrared and visual image fusion through feature extraction by morphological sequential toggle operator. Infrared Phys. Technol. 71, 77–86 (2015)

    Google Scholar 

  7. Bai, X., Zhou, F., Xue, B.: Edge preserved image fusion based on multiscale toggle contrast operator. Image Vis. Comput. 29(12), 829–839 (2011)

    Google Scholar 

  8. Bai, X., Chen, X., Zhou, F., et al.: Multiscale top-hat selection transform based infrared and visual image fusion with emphasis on extracting regions of interest. Infrared Phys. Technol. 60, 81–93 (2013)

    Google Scholar 

  9. Bai, X., Gu, S., Zhou, F., et al.: Weighted image fusion based on multi-scale top-hat transform: algorithms and a comparison study. Optik Int. J. Light Electron Opt. 124(13), 1660–1668 (2013)

    Google Scholar 

  10. Cao, L., Jin, L., Tao, H., et al.: Multi-focus image fusion based on spatial frequency in discrete cosine transform domain. IEEE Signal Process. Lett. 22(2), 220–224 (2014)

    Google Scholar 

  11. Chen, J., Li, X., Luo, L., et al.: Multi-focus image fusion based on multi-scale gradients and image matting. IEEE Trans. Multimed. 24, 655–667 (2022). https://doi.org/10.1109/TMM.2021.3057493

    Article  Google Scholar 

  12. Choi, J., Yu, K., Kim, Y.: A new adaptive component-substitution-based satellite image fusion by using partial replacement. IEEE Trans. Geosci. Remote Sens. 49(1), 295–309 (2010)

    Google Scholar 

  13. Choi, Y., Yang, D., Han, S., et al.: Change target extraction based on scale-adaptive difference image and morphology filter for kompsat-5. Remote Sens. 14(2), 245 (2022)

    Google Scholar 

  14. De, I., Chanda, B.: A simple and efficient algorithm for multifocus image fusion using morphological wavelets. Signal Process. 86(5), 924–936 (2006)

    Google Scholar 

  15. De, I., Chanda, B.: Multi-focus image fusion using a morphology-based focus measure in a quad-tree structure. Inf. Fus. 14(2), 136–146 (2013)

    Google Scholar 

  16. Dou, J., Li, J.: Image fusion quality assessment based on discrete cosine transform and human visual system. Opt. Eng. 51(9), 097,002 (2012)

    Google Scholar 

  17. Eskicioglu, A.M., Fisher, P.S.: Image quality measures and their performance. IEEE Trans. Commun. 43(12), 2959–2965 (1995)

    Google Scholar 

  18. Fang, C., Wang, L., Zhang, D., et al.: Incremental cross-view mutual distillation for self-supervised medical ct synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20,677–20,686 (2022)

  19. Fang, C., Zhang, D., Wang, L., et al.: Cross-modality high-frequency transformer for mr image super-resolution. arXiv preprint arXiv:2203.15314 (2022)

  20. Garzelli, A., Nencini, F.: Hypercomplex quality assessment of multi/hyperspectral images. IEEE Geosci. Remote Sens. Lett. 6(4), 662–665 (2009)

    Google Scholar 

  21. Gong, M., Ma, J., Xu, H., et al.: D2tnet: a convlstm network with dual-direction transfer for pan-sharpening. IEEE Trans. Geosci. Remote Sens. 60, 1–14 (2022)

    Google Scholar 

  22. Haghighat, M.B.A., Aghagolzadeh, A., Seyedarabi, H.: Multi-focus image fusion for visual sensor networks in dct domain. Comput. Electr. Eng. 37(5), 789–797 (2011)

    Google Scholar 

  23. Haghighat, M.B.A., Aghagolzadeh, A., Seyedarabi, H.: A non-reference image fusion metric based on mutual information of image features. Comput. Electr. Eng. 37(5), 744–756 (2011)

    Google Scholar 

  24. Hayat, N., Imran, M.: Ghost-free multi exposure image fusion technique using dense sift descriptor and guided filter. J. Vis. Commun. Image Represent. 62, 295–308 (2019)

    Google Scholar 

  25. Hermessi, H., Mourali, O., Zagrouba, E.: Multimodal medical image fusion review: theoretical background and recent advances. Signal Process. 183(108), 036 (2021)

    Google Scholar 

  26. Huang, Z., Liu, J., Fan, X., et al.: Reconet: recurrent correction network for fast and efficient multi-modality image fusion. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVIII, Springer, pp. 539–555 (2022)

  27. Jiang, Y., Wang, M.: Image fusion with morphological component analysis. Inf. Fus. 18, 107–118 (2014)

    Google Scholar 

  28. Jin, X., Jiang, Q., Yao, S., et al.: Infrared and visual image fusion method based on discrete cosine transform and local spatial frequency in discrete stationary wavelet transform domain. Infrared Phys. Technol. 88, 1–12 (2018)

    Google Scholar 

  29. Kaur, H., Koundal, D., Kadyan, V.: Image fusion techniques: a survey. Arch. Comput. Methods Eng. pp. 1–23 (2021)

  30. Kumar, B.S.: Multifocus and multispectral image fusion based on pixel significance using discrete cosine harmonic wavelet transform. SIViP 7(6), 1125–1143 (2013)

    Google Scholar 

  31. Kumar, B.S.: Image fusion based on pixel significance using cross bilateral filter. SIViP 9(5), 1193–1204 (2015)

    Google Scholar 

  32. Kumar, B.S., Swamy, M., Ahmad, M.O.: Multiresolution dct decomposition for multifocus image fusion. In: 2013 26th IEEE Canadian Conference on Electrical and Computer Engineering (CCECE). IEEE, pp. 1–4 (2013)

  33. Li, J., Liu, J., Zhou, S., et al.: Learning a coordinated network for detail-refinement multi-exposure image fusion. IEEE Trans. Circuits Syst. Video Technol. (2022)

  34. Li, J., Liu, J., Zhou, S., et al.: Infrared and visible image fusion based on residual dense network and gradient loss. Infrared Phys. Technol. 128(104), 486 (2023)

    Google Scholar 

  35. Li, W., Xie, Y., Zhou, H., et al.: Structure-aware image fusion. Optik 172, 1–11 (2018)

    Google Scholar 

  36. Liu, J., Fan, X., Jiang, J., et al.: Learning a deep multi-scale feature ensemble and an edge-attention guidance for image fusion. IEEE Trans. Circuits Syst. Video Technol. 32(1), 105–119 (2021)

    Google Scholar 

  37. Liu, J., Wu, Y., Huang, Z., et al.: Smoa: searching a modality-oriented architecture for infrared and visible image fusion. IEEE Signal Process. Lett. 28, 1818–1822 (2021)

    Google Scholar 

  38. Liu, J., Fan, X., Huang, Z., et al.: Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5802–5811 (2022)

  39. Liu, J., Jiang, Z., Wu, G., et al.: A unified image fusion framework with flexible bilevel paradigm integration. Vis. Comput, 1–18 (2022)

  40. Liu, R., Liu, J., Jiang, Z., et al.: A bilevel integrated model with data-driven layer ensemble for multi-modality image fusion. IEEE Trans. Image Process. 30, 1261–1274 (2020)

    Google Scholar 

  41. Liu, Y., Wang, Z.: A practical pan-sharpening method with wavelet transform and sparse representation. In: 2013 IEEE International Conference on Imaging Systems and Techniques (IST). IEEE, pp. 288–293 (2013)

  42. Liu, Y., Wang, Z.: Dense sift for ghost-free multi-exposure fusion. J. Vis. Commun. Image Represent. 31, 208–224 (2015)

    Google Scholar 

  43. Liu, Y., Liu, S., Wang, Z.: A general framework for image fusion based on multi-scale transform and sparse representation. Inf. Fus. 24, 147–164 (2015)

    Google Scholar 

  44. Liu, Y., Chen, X., Peng, H., et al.: Multi-focus image fusion with a deep convolutional neural network. Inf. Fus. 36, 191–207 (2017)

    Google Scholar 

  45. Ma, J., Ma, Y., Li, C.: Infrared and visible image fusion methods and applications: a survey. Inf. Fus. 45, 153–178 (2019)

    Google Scholar 

  46. Ma, J., Tang, L., Fan, F., et al.: Swinfusion: cross-domain long-range learning for general image fusion via swin transformer. IEEE/CAA J. Autom. Sin. 9(7), 1200–1217 (2022)

    Google Scholar 

  47. Matsopoulos, G.K., Marshall, S.: Application of morphological pyramids: fusion of mr and ct phantoms. J. Vis. Commun. Image Represent. 6(2), 196–207 (1995)

    Google Scholar 

  48. Nayak, A., et al.: A systematic exploration of image fusion: a review. ICDSMLA 2020, 1607–1613 (2022)

    Google Scholar 

  49. Nejati, M., Samavi, S., Shirani, S.: Multi-focus image fusion using dictionary-based sparse representation. Inf. Fus. 25, 72–84 (2015)

    Google Scholar 

  50. Nie, X., Xiao, B., Bi, X., et al.: A focus measure in discrete cosine transform domain for multi-focus image fast fusion. Neurocomputing 465, 93–102 (2021)

    Google Scholar 

  51. Paramanandham, N., Rajendiran, K.: Infrared and visible image fusion using discrete cosine transform and swarm intelligence for surveillance applications. Infrared Phys. Technol. 88, 13–22 (2018)

    Google Scholar 

  52. Paul, S., Sevcenco, I.S., Agathoklis, P.: Multi-exposure and multi-focus image fusion in gradient domain. J. Circuits Syst. Comput. 25(10), 1650,123 (2016)

    Google Scholar 

  53. Piella, G., Heijmans, H.: A new quality metric for image fusion. In: Proceedings 2003 International Conference on Image Processing (Cat. No. 03CH37429). IEEE, pp. 3–173 (2003)

  54. Purkait, P., Chanda, B.: Super resolution image reconstruction through Bregman iteration using morphologic regularization. IEEE Trans. Image Process. 21(9), 4029–4039 (2012)

    MathSciNet  Google Scholar 

  55. Ramlal, S.D., Sachdeva, J., Ahuja, C.K., et al.: Multimodal medical image fusion using non-subsampled Shearlet transform and pulse coupled neural network incorporated with morphological gradient. SIViP 12(8), 1479–1487 (2018)

    Google Scholar 

  56. Redmon, J., Divvala, S., Girshick, R., et al: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779–788 (2016)

  57. Restaino, R., Vivone, G., Dalla Mura, M., et al.: Fusion of multispectral and panchromatic images based on morphological operators. IEEE Trans. Image Process. 25(6), 2882–2895 (2016)

    MathSciNet  Google Scholar 

  58. Schulze, M., Pearce, J.: Linear combinations of morphological operators: the midrange, pseudomedian, and loco filters. In: 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 5, pp. 57–60 (1993). https://doi.org/10.1109/ICASSP.1993.319746

  59. Tang, J.: A contrast based image fusion technique in the dct domain. Digital Signal Process. 14(3), 218–226 (2004)

  60. Tang, L., Yuan, J., Zhang, H., et al.: Piafusion: a progressive infrared and visible image fusion network based on illumination aware. Inf. Fus. 83, 79–92 (2022)

    Google Scholar 

  61. Toet, A.: The tno multiband image data collection. Data Brief 15, 249–251 (2017)

    Google Scholar 

  62. Wald, L.: Data fusion: definitions and architectures: fusion of images of different spatial resolutions. Presses des MINES (2002)

  63. Wang, D., Liu, J., Fan, X., et al.: Unsupervised misaligned infrared and visible image fusion via cross-modality image generation and registration. arXiv preprint arXiv:2205.11876 (2022)

  64. Wang, R., Fang, N., He, Y., et al.: Multi-modal medical image fusion based on geometric algebra discrete cosine transform. Adv. Appl. Clifford Algebras 32(2), 1–23 (2022)

    MathSciNet  Google Scholar 

  65. Wang, Z., Bovik, A.C.: A universal image quality index. IEEE Signal Process. Lett. 9(3), 81–84 (2002)

    Google Scholar 

  66. Xu, H., Ma, J., Jiang, J., et al.: U2fusion: a unified unsupervised image fusion network. IEEE Trans. Pattern Anal. Mach. Intell. 44(1), 502–518 (2020)

    Google Scholar 

  67. Xu, H., Ma, J., Yuan, J., et al.: Rfnet: unsupervised network for mutually reinforcing multi-modal image registration and fusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19,679–19,688 (2022)

  68. Xu, Z.: Medical image fusion using multi-level local extrema. Inf. Fus. 19, 38–48 (2014)

    Google Scholar 

  69. Xydeas, C., Petrovic, V.: Objective image fusion performance measure. Electron. Lett. 36(4), 308–309 (2000)

    Google Scholar 

  70. Yang, B., Jing, Z.: Medical image fusion with a shift-invariant morphological wavelet. In: 2008 IEEE Conference on Cybernetics and Intelligent Systems. IEEE, pp. 175–178 (2008)

  71. Yilmaz, C.S., Yilmaz, V., Gungor, O.: A theoretical and practical survey of image fusion methods for multispectral pansharpening. Inf. Fus. 79, 1–43 (2022)

    Google Scholar 

  72. Zafar, I., Edirisinghe, E., Bez, H.: Multi-exposure and Multi-focus Image Fusion in Transform Domain. IET, Stevenage (2006)

    Google Scholar 

  73. Zhan, K., Xie, Y., Wang, H., et al.: Fast filtering image fusion. J. Electron. Imaging 26(6), 063,004 (2017)

    Google Scholar 

  74. Zhang, H., Xu, H., Xiao, Y., et al.: Rethinking the image fusion: a fast unified image fusion network based on proportional maintenance of gradient and intensity. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 12,797–12,804 (2020)

  75. Zhang, H., Le, Z., Shao, Z., et al.: Mff-gan: an unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion. Inf. Fus. 66, 40–53 (2021)

  76. Zhang, H., Xu, H., Tian, X., et al.: Image fusion meets deep learning: a survey and perspective. Inf. Fus. 76, 323–336 (2021)

    Google Scholar 

  77. Zhang, X.: Benchmarking and comparing multi-exposure image fusion algorithms. Inf. Fus. 74, 111–131 (2021)

    Google Scholar 

  78. Zhang, Y., Bai, X., Wang, T.: Multi-focus image fusion via boundary finding and multi-scale morphological focus-measure. In: 2014 International Conference on Digital Image Computing: Techniques and Applications (DICTA). IEEE, pp. 1–7 (2014)

  79. Zhang, Y., Liu, Y., Sun, P., et al.: Ifcnn: a general image fusion framework based on convolutional neural network. Inf. Fus. 54, 99–118 (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of CSE, Indian Institute of Technology (ISM), Dhanbad, Jharkhand, 826004, India

    Manali Roy & Susanta Mukhopadhyay

Authors
  1. Manali Roy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Susanta Mukhopadhyay
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Manali Roy.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Roy, M., Mukhopadhyay, S. A DCT-based multiscale framework for 2D greyscale image fusion using morphological differential features. Vis Comput 40, 3569–3590 (2024). https://doi.org/10.1007/s00371-023-03052-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-03052-0

Keywords

Search

Navigation