Abstract
This research discusses the pre-trained deep learning architecture for the multimodal learning and representation in surveillance system. This framework generates a single image from the integration of the multi sensor information, which includes the infrared and visible. We use visible and infrared as the information in different spectrum of light, in term of contrast ratio and visibility. We start with image registration to align coordinates so the decomposition of the source image into the sub bands is possible. The VGG-19 and the weighted averaging are utilized for the feature extraction and transfer learning task. This is conducted thorough empirical research by implementing a series of methodology studies to evaluate how pre-trained deep learning techniques enhance overall fusion performance and improve recognition and detection capability. This study also contains a comparison of the performance of spatial and frequency algorithms in contrast to the deep learning based method for the surveillance system. The research work is concluded by evaluating the performance measure of the proposed fusion algorithm with the traditional algorithm.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Paramanandham, N., Rajendiran, K.: Multi sensor image fusion for surveillance applications using hybrid image fusion algorithm. Multimedia Tools and Applications 77(10), 12405–12436 (2017). https://doi.org/10.1007/s11042-017-4895-3
Azam, M.A., et al.: A review on multimodal medical image fusion: compendious analysis of medical modalities, multimodal databases, fusion techniques and quality metrics. Comput. Biol. Med. 144, 105253 (2018)
Tang, L., Yuan, J., Ma, J.: Image fusion in the loop of high-level vision tasks: a semantic-aware real-time infrared and visible image fusion network. Inf. Fusion 82, 28–42 (2022)
Alseelawi, N., Hazim, H.T., Salim ALRikabi, H.T.: A Novel method of multimodal medical image fusion based on hybrid approach of NSCT and DTCWT. Int. J. Online Biomed. Eng. 18(3) (2022)
Zhang, H., Xu, H., Tian, X., Jiang, J., Ma, J.: Image fusion meets deep learning: a survey and perspective. Inf. Fusion 76, 323–336 (2021)
Kaur, H., Koundal, D., Kadyan, V.: Image fusion techniques: a survey. Arch. Comput. Methods Eng. 28(7), 4425–4447 (2021). https://doi.org/10.1007/s11831-021-09540-7
Huang, B., Yang, F., Yin, M., Mo, X., Zhong, C.: A review of multimodal medical image fusion techniques. Comput. Math. Methods Med. 144 (2020)
Ma, J, .Ma, Y., Li, C.: Infrared and visible image fusion methods and applications: a survey. Inf. Fusion. 45, 153–178 (2019). https://doi.org/10.1016/j.inffus.2018.02.004
Ma, J., Tang, L., Fan, F., Huang, J., Mei, X., Ma, Y.: SwinFusion: cross-domain long-range learning for general image fusion via swin transformer. IEEE/CAA J. Automat. Sin. 9(7), 1200–1217 (2022)
Hermessi, H., Mourali, O., Zagrouba, E.: Multimodal medical image fusion review: theoretical background and recent advances. Signal Process. 183, 108036 (2021)
Li, Y., Zhao, J., Lv, Z., Li, J.: Medical image fusion method by deep learning. Int. J. Cogni. Comput. Eng. 2, 21–29 (2021)
Li, G., Lin, Y., Qu, X.: An infrared and visible image fusion method based on multi-scale transformation and norm optimization. Inf. Fusion 71, 109–129 (2021)
Xu, H., Ma, J., Jiang, J., Guo, X., Ling, H.: U2Fusion: a unified unsupervised image fusion network. IEEE Trans. Pattern Anal. Mach. Intell. 44(1), 502–518 (2020)
Anandhi, D., Valli, S.: An algorithm for multi-sensor image fusion using maximum a posteriori and nonsubsampled contourlet transform. Comput. Electr. Eng. 65, 139–152 (2018)
Cai, J., Cheng, Q., Peng, M., Song, Y.: Fusion of infrared and visible images based on nonsubsampled contourlet transform and sparse K-SVD dictionary learning. Infrared Phys. Technol. 82, 85–95 (2017)
Tong, Y.: Visual sensor image enhancement based on non-sub-sampled shearlet transform and phase stretch transform. EURASIP J. Wirel. Commun. Netw. 2019(1), 1–8 (2019). https://doi.org/10.1186/s13638-019-1344-1
Wang, Z., Ziou, D., Armenakis, C., Li, D., Li, Q.: A comparative analysis of image fusion methods. IEEE Trans. Geosci. Remote Sens. 43(6), 1391–1402 (2005)
Pajares, G., De La Cruz, J.M.: A wavelet-based image fusion tutorial. Pattern Recogn. 37(9), 1855–1872 (2004)
Li, S., Kang, X., Hu, J.: Image fusion with guided filtering. IEEE Trans. Image Process. 22(7), 2864–2875 (2013)
Li, S., Kang, X., Fang, L., Hu, J., Yin, H.: Pixel-level image fusion: a survey of the state of the art. Inf. Fusion 33, 100–112 (2017)
Zhang, Y., Liu, Y., Sun, P., Yan, H., Zhao, X., Zhang, L.: IFCNN: a general image fusion framework based on convolutional neural network. Inf. Fusion 54, 99–118 (2020)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Chib, P.S., Khari, M., Santosh, K. (2023). A Computational Study on Calibrated VGG19 for Multimodal Learning and Representation in Surveillance. In: Santosh, K., Goyal, A., Aouada, D., Makkar, A., Chiang, YY., Singh, S.K. (eds) Recent Trends in Image Processing and Pattern Recognition. RTIP2R 2022. Communications in Computer and Information Science, vol 1704. Springer, Cham. https://doi.org/10.1007/978-3-031-23599-3_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-23599-3_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-23598-6
Online ISBN: 978-3-031-23599-3
eBook Packages: Computer ScienceComputer Science (R0)