Abstract
Medical image fusion has been shown to be effective in supporting clinicians make better clinical diagnoses. Although many algorithms have been proposed for synthesis, they still have certain limitations. Some limitations can be seen as the synthesized image is reduced in contrast or details are not preserved. In this paper, we propose an image fusion algorithm to solve the problems mentioned above. Firstly, an image decomposition method is proposed to decompose the image into two components. This method is based on the Gaussian filter and the Weighted mean curvature filter. Secondly, a fusion method for high-frequency components is based on local energy function using Structure tensor saliency. Finally, we create an adaptive fusion rule using the Marine Predators Algorithm optimization method to fuse low-frequency components. Five latest algorithms and five evaluation indexes have been used to test the proposed algorithm’s effectiveness. The obtained experimental results show that the composite image is significantly improved in quality as well as well preserved the information from the input image.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abdel-Basset, M., Mohamed, R., Elhoseny, M., Chakrabortty, R. K., & Ryan, M. (2020). A hybrid COVID-19 detection model using an improved marine predators algorithm and a ranking-based diversity reduction strategy. IEEE Access, 8, 79521–79540. https://doi.org/10.1109/access.2020.2990893.
Barba-J, L., Vargas-Quintero, L., & Calderón-Agudelo, J. A. (2022). Bone SPECT/CT image fusion based on the discrete hermite transform and sparse representation. Biomedical Signal Processing and Control, 71, 103096. https://doi.org/10.1016/j.bspc.2021.103096.
Daniel, E., Anitha, J., Kamaleshwaran, K., & Rani, I. (2017). Optimum spectrum mask based medical image fusion using gray wolf optimization. Biomedical Signal Processing and Control, 34, 36–43. https://doi.org/10.1016/j.bspc.2017.01.003.
Ding, Z., Zhou, D., Nie, R., Hou, R., & Liu, Y. (2020). Brain medical image fusion based on dual-branch CNNs in NSST domain. BioMed Research International, 2020, 1–15. https://doi.org/10.1155/2020/6265708.
Ding, Z., Zhou, D., Li, H., Hou, R., & Liu, Y. (2021). Siamese networks and multi-scale local extrema scheme for multimodal brain medical image fusion. Biomedical Signal Processing and Control, 68, 102697. https://doi.org/10.1016/j.bspc.2021.102697.
Dinh, P. H. (2021). Combining Gabor energy with equilibrium optimizer algorithm for multi-modality medical image fusion. Biomedical Signal Processing and Control, 68, 102696. https://doi.org/10.1016/j.bspc.2021.102696.
Dinh, P. H. (2021). An improved medical image synthesis approach based on marine predators algorithm and maximum Gabor energy. Neural Computing and Applications. https://doi.org/10.1007/s00521-021-06577-4.
Dinh, P. H. (2021). Multi-modal medical image fusion based on equilibrium optimizer algorithm and local energy functions. Applied Intelligence. https://doi.org/10.1007/s10489-021-02282-w.
Dinh, P. H. (2021). A novel approach based on grasshopper optimization algorithm for medical image fusion. Expert Systems with Applications, 171, 114576. https://doi.org/10.1016/j.eswa.2021.114576.
Dinh, P. H. (2021). A novel approach based on three-scale image decomposition and marine predators algorithm for multi-modal medical image fusion. Biomedical Signal Processing and Control, 67, 102536. https://doi.org/10.1016/j.bspc.2021.102536.
Du, J., Li, W., Xiao, B., & Nawaz, Q. (2016). Union Laplacian pyramid with multiple features for medical image fusion. Neurocomputing, 194, 326–339. https://doi.org/10.1016/j.neucom.2016.02.047.
Du, J., Li, W., & Tan, H. (2020). Three-layer image representation by an enhanced illumination-based image fusion method. IEEE Journal of Biomedical and Health Informatics, 24(4), 1169–1179. https://doi.org/10.1109/jbhi.2019.2930978.
Fang, H., Zhu, G., Stojanovic, V., Nie, R., He, S., Luan, X., & Liu, F. (2021). Adaptive optimization algorithm for nonlinear Markov jump systems with partial unknown dynamics. International Journal of Robust and Nonlinear Control, 31(6), 2126–2140. https://doi.org/10.1002/rnc.5350.
Faramarzi, A., Heidarinejad, M., Mirjalili, S., & Gandomi, A. H. (2020). Marine predators algorithm: A nature-inspired metaheuristic. Expert Systems with Applications, 152, 113377. https://doi.org/10.1016/j.eswa.2020.113377.
Fu, J., Li, W., Du, J., & Xiao, B. (2020). Multimodal medical image fusion via Laplacian pyramid and convolutional neural network reconstruction with local gradient energy strategy. Computers in Biology and Medicine, 126, 104048. https://doi.org/10.1016/j.compbiomed.2020.104048.
Gao, Y., Ma, S., Liu, J., Liu, Y., & Zhang, X. (2021). Fusion of medical images based on salient features extraction by PSO optimized fuzzy logic in NSST domain. Biomedical Signal Processing and Control, 69, 102852. https://doi.org/10.1016/j.bspc.2021.102852.
Gong, Y., & Goksel, O. (2019). Weighted mean curvature. Signal Processing, 164, 329–339. https://doi.org/10.1016/j.sigpro.2019.06.020.
Guo, K., Hu, X., & Li, X. (2021). MMFGAN: A novel multimodal brain medical image fusion based on the improvement of generative adversarial network. Multimedia Tools and Applications. https://doi.org/10.1007/s11042-021-11822-y.
Haghighat, M. B. A., Aghagolzadeh, A., & Seyedarabi, H. (2011). A non-reference image fusion metric based on mutual information of image features. Computers & Electrical Engineering, 37, 744–756. https://doi.org/10.1016/j.compeleceng.2011.07.012.
Han, Y., Cai, Y., Cao, Y., & Xu, X. (2013). A new image fusion performance metric based on visual information fidelity. Information Fusion, 14, 127–135. https://doi.org/10.1016/j.inffus.2011.08.002.
Hu, Q., Hu, S., & Zhang, F. (2021). Multi-modality image fusion combining sparse representation with guidance filtering. Soft Computing, 25(6), 4393–4407. https://doi.org/10.1007/s00500-020-05448-9.
Jose, J., Gautam, N., Tiwari, M., Tiwari, T., Suresh, A., Sundararaj, V., & Rejeesh, M. R. (2021). An image quality enhancement scheme employing adolescent identity search algorithm in the NSST domain for multimodal medical image fusion. Biomedical Signal Processing and Control, 66, 102480. https://doi.org/10.1016/j.bspc.2021.102480.
Li, Q., Wang, W., Chen, G., & Zhao, D. (2021). Medical image fusion using segment graph filter and sparse representation. Computers in Biology and Medicine, 131, 104239. https://doi.org/10.1016/j.compbiomed.2021.104239.
Li, W., Peng, X., Fu, J., Wang, G., Huang, Y., & Chao, F. (2021). A multiscale double-branch residual attention network for anatomical-functional medical image fusion. Computers in Biology and Medicine, 141, 105005. https://doi.org/10.1016/j.compbiomed.2021.105005.
Li, X., Zhang, X., & Ding, M. (2019). A sum-modified-Laplacian and sparse representation based multimodal medical image fusion in Laplacian pyramid domain. Medical & Biological Engineering & Computing, 57(10), 2265–2275. https://doi.org/10.1007/s11517-019-02023-9.
Li, X., Zhou, F., Tan, H., Zhang, W., & Zhao, C. (2021). Multimodal medical image fusion based on joint bilateral filter and local gradient energy. Information Sciences, 569, 302–325. https://doi.org/10.1016/j.ins.2021.04.052.
Li, Y., Sun, Y., Huang, X., Qi, G., Zheng, M., & Zhu, Z. (2018). An image fusion method based on sparse representation and sum modified-Laplacian in NSCT domain. Entropy, 20(7), 522. https://doi.org/10.3390/e20070522.
Liu, X., Mei, W., & Du, H. (2018). Multi-modality medical image fusion based on image decomposition framework and nonsubsampled Shearlet transform. Biomedical Signal Processing and Control, 40, 343–350. https://doi.org/10.1016/j.bspc.2017.10.001.
Liu, Y., Chen, X., Ward, R. K., & Wang, Z. J. (2016). Image fusion with convolutional sparse representation. IEEE Signal Processing Letters, 23(12), 1882–1886. https://doi.org/10.1109/lsp.2016.2618776.
Liu, Y., Chen, X., Ward, R. K., & Wang, Z. J. (2019). Medical image fusion via convolutional sparsity based morphological component analysis. IEEE Signal Processing Letters, 26, 485–489. https://doi.org/10.1109/lsp.2019.2895749.
Maqsood, S., & Javed, U. (2020). Multi-modal medical image fusion based on two-scale image decomposition and sparse representation. Biomedical Signal Processing and Control, 57, 101810. https://doi.org/10.1016/j.bspc.2019.101810.
Mirjalili, S. (2015). The ant lion optimizer. Advances in Engineering Software, 83, 80–98. https://doi.org/10.1016/j.advengsoft.2015.01.010.
Mirjalili, S. (2016). SCA: A sine cosine algorithm for solving optimization problems. Knowledge-Based Systems, 96, 120–133. https://doi.org/10.1016/j.knosys.2015.12.022.
Mirjalili, S., & Lewis, A. (2016). The whale optimization algorithm. Advances in Engineering Software, 95, 51–67. https://doi.org/10.1016/j.advengsoft.2016.01.008.
Mirjalili, S., Mirjalili, S. M., & Hatamlou, A. (2015). Multi-verse optimizer: A nature-inspired algorithm for global optimization. Neural Computing and Applications, 27(2), 495–513. https://doi.org/10.1007/s00521-015-1870-7.
Mirjalili, S., Gandomi, A. H., Mirjalili, S. Z., Saremi, S., Faris, H., & Mirjalili, S. M. (2017). Salp swarm algorithm: A bio-inspired optimizer for engineering design problems. Advances in Engineering Software, 114, 163–191. https://doi.org/10.1016/j.advengsoft.2017.07.002.
Nair, R. R., & Singh, T. (2021). MAMIF: Multimodal adaptive medical image fusion based on b-spline registration and non-subsampled shearlet transform. Multimedia Tools and Applications, 80(12), 19079–19105. https://doi.org/10.1007/s11042-020-10439-x.
Nair, R. R., & Singh, T. (2021). An optimal registration on shearlet domain with novel weighted energy fusion for multi-modal medical images. Optik, 225, 165742. https://doi.org/10.1016/j.ijleo.2020.165742.
Polinati, S., & Dhuli, R. (2020). Multimodal medical image fusion using empirical wavelet decomposition and local energy maxima. Optik, 205, 163947. https://doi.org/10.1016/j.ijleo.2019.163947.
Pršić, D., Nedić, N., & Stojanović, V. (2016). A nature inspired optimal control of pneumatic-driven parallel robot platform. Proceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering Science, 231(1), 59–71. https://doi.org/10.1177/0954406216662367.
Ren, Z., Guan, P., Lam, E. Y., & Zhao, J. (2021). Extended focused imaging in microscopy using structure tensor and guided filtering. Optics and Lasers in Engineering, 140, 106549. https://doi.org/10.1016/j.optlaseng.2021.106549.
Shehanaz, S., Daniel, E., Guntur, S. R., & Satrasupalli, S. (2021). Optimum weighted multimodal medical image fusion using particle swarm optimization. Optik, 231, 166413. https://doi.org/10.1016/j.ijleo.2021.166413.
Shibu, D. S., & Priyadharsini, S. S. (2021). Multi scale decomposition based medical image fusion using convolutional neural network and sparse representation. Biomedical Signal Processing and Control, 69, 102789. https://doi.org/10.1016/j.bspc.2021.102789.
Stojanovic, V., He, S., & Zhang, B. (2020). State and parameter joint estimation of linear stochastic systems in presence of faults and non-Gaussian noises. International Journal of Robust and Nonlinear Control, 30(16), 6683–6700. https://doi.org/10.1002/rnc.5131.
Tan, W., Tiwari, P., Pandey, H. M., Moreira, C., & Jaiswal, A. K. (2020). Multimodal medical image fusion algorithm in the era of big data. Neural Computing and Applications. https://doi.org/10.1007/s00521-020-05173-2.
Tan, W., Thitøn, W., Xiang, P., & Zhou, H. (2021). Multi-modal brain image fusion based on multi-level edge-preserving filtering. Biomedical Signal Processing and Control, 64, 102280. https://doi.org/10.1016/j.bspc.2020.102280.
Tannaz, A., Mousa, S., Sabalan, D., & Masoud, P. (2019). Fusion of multimodal medical images using nonsubsampled shearlet transform and particle swarm optimization. Multidimensional Systems and Signal Processing, 31(1), 269–287. https://doi.org/10.1007/s11045-019-00662-7.
Tao, H., Li, J., Chen, Y., Stojanovic, V., & Yang, H. (2020). Robust point-to-point iterative learning control with trial-varying initial conditions. IET Control Theory & Applications, 14(19), 3344–3350. https://doi.org/10.1049/iet-cta.2020.0557.
Wang, G., Li, W., & Huang, Y. (2021). Medical image fusion based on hybrid three-layer decomposition model and nuclear norm. Computers in Biology and Medicine, 129, 104179. https://doi.org/10.1016/j.compbiomed.2020.104179.
Wang, L., Dou, J., Qin, P., Lin, S., Gao, Y., Wang, R., & Zhang, J. (2021). Multimodal medical image fusion based on nonsubsampled shearlet transform and convolutional sparse representation. Multimedia Tools and Applications, 80(30), 36401–36421. https://doi.org/10.1007/s11042-021-11379-w.
Wang, M., Zhang, W., Sun, C., & Sowmya, A. (2020). Corner detection based on shearlet transform and multi-directional structure tensor. Pattern Recognition, 103, 107299. https://doi.org/10.1016/j.patcog.2020.107299.
Wang, S., & Shen, Y. (2020). Multi-modal image fusion based on saliency guided in NSCT domain. IET Image Processing. https://doi.org/10.1049/iet-ipr.2019.1319.
Wang, S., Celebi, M. E., Zhang, Y. D., Yu, X., Lu, S., Yao, X., et al. (2021). Advances in data preprocessing for biomedical data fusion: An overview of the methods, challenges, and prospects. Information Fusion, 76, 376–421. https://doi.org/10.1016/j.inffus.2021.07.001.
Wang, Z., Cui, Z., & Zhu, Y. (2020). Multi-modal medical image fusion by Laplacian pyramid and adaptive sparse representation. Computers in Biology and Medicine, 123, 103823. https://doi.org/10.1016/j.compbiomed.2020.103823.
Wang, Z., Li, X., Duan, H., Su, Y., Zhang, X., & Guan, X. (2021). Medical image fusion based on convolutional neural networks and non-subsampled contourlet transform. Expert Systems with Applications, 171, 114574. https://doi.org/10.1016/j.eswa.2021.114574.
Wei, T., Li, X., & Stojanovic, V. (2021). Input-to-state stability of impulsive reaction–diffusion neural networks with infinite distributed delays. Nonlinear Dynamics, 103(2), 1733–1755. https://doi.org/10.1007/s11071-021-06208-6.
Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin, 1(6), 80. https://doi.org/10.2307/3001968.
Xing, C., Wang, M., Dong, C., Duan, C., & Wang, Z. (2020). Using Taylor expansion and convolutional sparse representation for image fusion. Neurocomputing, 402, 437–455. https://doi.org/10.1016/j.neucom.2020.04.002.
Xu, L., Si, Y., Jiang, S., Sun, Y., & Ebrahimian, H. (2020). Medical image fusion using a modified shark smell optimization algorithm and hybrid wavelet-homomorphic filter. Biomedical Signal Processing and Control, 59, 101885. https://doi.org/10.1016/j.bspc.2020.101885.
Xydeas, C., & Petrovic, V. (2000). Objective image fusion performance measure. Electronics Letters, 36, 308. https://doi.org/10.1049/el:20000267.
Yin, M., Liu, X., Liu, Y., & Chen, X. (2019). Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain. IEEE Transactions on Instrumentation and Measurement, 68, 49–64. https://doi.org/10.1109/tim.2018.2838778.
Yousif, A. S., Omar, Z., & Sheikh, U. U. (2022). An improved approach for medical image fusion using sparse representation and Siamese convolutional neural network. Biomedical Signal Processing and Control, 72, 103357. https://doi.org/10.1016/j.bspc.2021.103357.
Zhang, Y. D., Dong, Z., Wang, S. H., Yu, X., Yao, X., Zhou, Q., et al. (2020). Advances in multimodal data fusion in neuroimaging: Overview, challenges, and novel orientation. Information Fusion, 64, 149–187. https://doi.org/10.1016/j.inffus.2020.07.006.
Zhao, C., Wang, T., & Lei, B. (2020). Medical image fusion method based on dense block and deep convolutional generative adversarial network. Neural Computing and Applications, 33(12), 6595–6610. https://doi.org/10.1007/s00521-020-05421-5.
Zhu, Z., Zheng, M., Qi, G., Wang, D., & Xiang, Y. (2019). A phase congruency and local Laplacian energy based multi-modality medical image fusion method in NSCT domain. IEEE Access, 7, 20811–20824. https://doi.org/10.1109/access.2019.2898111.
Acknowledgements
This research is funded by Thuyloi University Foundation for Science and Technology under grant number TLU.STF.21-03.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Dinh, PH. A novel approach using structure tensor for medical image fusion. Multidim Syst Sign Process 33, 1001–1021 (2022). https://doi.org/10.1007/s11045-022-00829-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11045-022-00829-9