Abstract
Content-Based Image Retrieval (CBIR) leveraging semantic segmentation integrates semantic understanding with image retrieval, enabling users to search for images based on specific objects or regions within them. This paper presents a methodology for constructing image signatures, a pivotal element in enhancing image representation within a CBIR system. The efficiency and effectiveness of a CBIR system significantly hinge on the quality of the image signature, which serves as a compact and informative representation of raw image data. Our proposed methodology begins with emphasizing clear object or region boundaries through pixel-level semantic segmentation masks. A pretrained semantic segmentation model, such as DeepLab v3+, is employed to generate pixel-wise object class predictions, yielding the necessary segmentation masks. Subsequently, each image is segmented into meaningful regions based on these masks, and relevant features are extracted from each segmented region using a pre-trained Deep Convolutional Neural Network (DCNN) models AlexNet, VGG16 and ResNet-18. During the retrieval phase, when a user queries the system with an image, the query image is segmented using the pre-trained semantic segmentation model, and features are extracted from the segmented regions of the query image. These query features are utilized to search the database for the most similar regions or images. Similarity scores, calculated using Euclidean distance, are used to rank the database entries based on their similarity to the query, allowing for efficient retrieval of the top-k most similar regions or images. We found that for some classes semantic segmented based retrieval better performance in comparison to image based.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Camvid - the Cambridge-driving labeled video database. http://mi.eng.cam.ac.uk/research/projects/VideoRec/CamVid/
Alzu’bi, A., Amira, A., Ramzan, N.: Semantic content-based image retrieval: a comprehensive study. J. Vis. Commun. Image Represent. 32, 20–54 (2015)
Babenko, A., Slesarev, A., Chigorin, A., Lempitsky, V.: Neural codes for image retrieval. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision – ECCV 2014. ECCV 2014. LNCS, vol. 8689, pp. 584–599. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_38
Bronstein, A.M., Bronstein, M.M., Guibas, L.J., Ovsjanikov, M.: Shape google: geometric words and expressions for invariant shape retrieval. ACM Trans. Graph. (TOG) 30(1), 1–20 (2011)
Carneiro, G., Chan, A.B., Moreno, P.J., Vasconcelos, N.: Supervised learning of semantic classes for image annotation and retrieval. IEEE Trans. Pattern Anal. Mach. Intell. 29(3), 394–410 (2007)
Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 801–818 (2018)
Flickner, M., et al.: Query by image and video content: the QBIC system. Computer 28(9), 23–32 (1995)
Gurrin, C., Smeaton, A.F., Doherty, A.R., et al.: Lifelogging: personal big data. Found. Trends Inf. Retr. 8(1), 1–125 (2014)
Haralick, R.M., Shanmugam, K., Dinstein, I.H.: Textural features for image classification. IEEE Trans. Syst. Man Cybern. 6, 610–621 (1973)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Huang, J., Kumar, S.R., Mitra, M., Zhu, W.J., Zabih, R.: Image indexing using color correlograms. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 762–768. IEEE (1997)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25 (2012)
Kumar, S., Singh, M.K., Mishra, M.: Efficient deep feature based semantic image retrieval. Neural Process. Lett. 1–24 (2023)
Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., Alsaadi, F.E.: A survey of deep neural network architectures and their applications. Neurocomputing 234, 11–26 (2017)
Mezzoudj, S., Behloul, A., Seghir, R., Saadna, Y.: A parallel content-based image retrieval system using spark and tachyon frameworks. J. King Saud Univ.-Comput. Inf. Sci. 33(2), 141–149 (2021)
Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002)
Pentland, A., Picard, R.W., Sclaroff, S.: Photobook: content-based manipulation of image databases. Int. J. Comput. Vis. 18(3), 233–254 (1996)
Rui, Y., Huang, T.S., Chang, S.F.: Image retrieval: current techniques, promising directions, and open issues. J. Vis. Commun. Image Represent. 10(1), 39–62 (1999)
Shakarami, A., Tarrah, H.: An efficient image descriptor for image classification and CBIR. Optik 214, 164833 (2020)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Singha, M., Hemachandran, K.: Content based image retrieval using color and texture. Signal Image Process. 3(1), 39 (2012)
Smeulders, A.W., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell. 22(12), 1349–1380 (2000)
Tzelepi, M., Tefas, A.: Relevance feedback in deep convolutional neural networks for content based image retrieval. In: Proceedings of the 9th Hellenic Conference on Artificial Intelligence, pp. 1–7 (2016)
Xu, X., Lu, H., Song, J., Yang, Y., Shen, H.T., Li, X.: Ternary adversarial networks with self-supervision for zero-shot cross-modal retrieval. IEEE Trans. Cybern. 50(6), 2400–2413 (2019)
Zhu, L., Jin, H., Zheng, R., Feng, X.: Weighting scheme for image retrieval based on bag-of-visual-words. IET Image Proc. 8(9), 509–518 (2014)
Acknowledgment
This work is supported under Institute of Eminence(IoE) grant of Banaras Hindu University (B.H.U), Varanasi, India.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kumar, S., Singh, M., Ruchilekha, Singh, M.K. (2024). Semantic Segmentation Based Image Signature Generation for CBIR. In: Choi, B.J., Singh, D., Tiwary, U.S., Chung, WY. (eds) Intelligent Human Computer Interaction. IHCI 2023. Lecture Notes in Computer Science, vol 14532. Springer, Cham. https://doi.org/10.1007/978-3-031-53830-8_33
Download citation
DOI: https://doi.org/10.1007/978-3-031-53830-8_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-53829-2
Online ISBN: 978-3-031-53830-8
eBook Packages: Computer ScienceComputer Science (R0)